{"id":13468661,"url":"https://github.com/marsupialtail/quokka","last_synced_at":"2025-10-07T08:08:02.397Z","repository":{"id":37983782,"uuid":"429630067","full_name":"marsupialtail/quokka","owner":"marsupialtail","description":"Making data lake work for time series","archived":false,"fork":false,"pushed_at":"2024-08-21T01:49:07.000Z","size":40318,"stargazers_count":1183,"open_issues_count":12,"forks_count":61,"subscribers_count":13,"default_branch":"master","last_synced_at":"2025-10-05T06:53:12.609Z","etag":null,"topics":["data-lake-analytics","distributed","etl-framework","mlops","sql"],"latest_commit_sha":null,"homepage":"https://marsupialtail.github.io/quokka/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/marsupialtail.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-11-19T01:17:06.000Z","updated_at":"2025-09-05T16:32:33.000Z","dependencies_parsed_at":"2023-02-14T09:47:04.565Z","dependency_job_id":"6dad24c5-8892-49a9-bf85-a8fa15d48e80","html_url":"https://github.com/marsupialtail/quokka","commit_stats":{"total_commits":597,"total_committers":19,"mean_commits":31.42105263157895,"dds":0.4623115577889447,"last_synced_commit":"1caf62e8a723a4b9117880857d1ccaa75672f315"},"previous_names":[],"tags_count":10,"template":false,"template_full_name":null,"purl":"pkg:github/marsupialtail/quokka","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marsupialtail%2Fquokka","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marsupialtail%2Fquokka/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marsupialtail%2Fquokka/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marsupialtail%2Fquokka/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/marsupialtail","download_url":"https://codeload.github.com/marsupialtail/quokka/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/marsupialtail%2Fquokka/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":278740839,"owners_count":26037481,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-07T02:00:06.786Z","response_time":59,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-lake-analytics","distributed","etl-framework","mlops","sql"],"created_at":"2024-07-31T15:01:16.062Z","updated_at":"2025-10-07T08:08:02.382Z","avatar_url":"https://github.com/marsupialtail.png","language":"Python","funding_links":[],"categories":["Python","Models and Projects"],"sub_categories":["Ray + Database"],"readme":"\u003cdiv align=\"center\"\u003e\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/marsupialtail/quokka/blob/master/docs/quokka-banner.png?raw=true\" alt=\"Title\"/\u003e\n\n**Making data lakes work for time series.\u003cbr/\u003e**\n**Built on Python with [DuckDB](https://github.com/duckdb/duckdb), [Polars](https://github.com/pola-rs/polars), [Ray](https://github.com/ray-project/ray), [Arrow](https://github.com/apache/arrow), [Redis](https://github.com/redis/redis) and [SQLGlot](https://github.com/tobymao/sqlglot).\u003cbr/\u003e**\n# **Quokka's Focus: [New Blog Post](https://github.com/marsupialtail/quokka/blob/master/blog/time.md) \u003cbr/\u003e**\n\n\u003ca href=\"https://discord.gg/6ujVV9HAg3\" style=\"display:inline-block;\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/-Join%20Quokka%20Discord-blue?logo=discord\" alt=\"Join Discord\" height=\"25px\"/\u003e\n\u003c/a\u003e\n\u003ca href=\"https://marsupialtail.github.io/quokka/\"\u003e\n    \u003cimg src=\"https://github.com/marsupialtail/quokka/blob/master/docs/docs/badge.svg\" alt=\"rust docs\" height=\"25px\"/\u003e\n\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/pyquokka/\"\u003e\n    \u003cimg src=\"https://img.shields.io/pypi/v/pyquokka.svg\" alt=\"PyPi Latest Release\" height=\"25px\"/\u003e\n\u003c/a\u003e\n\u003c/p\u003e\n\u003c/div\u003e\n\nIf you would like to support Quokka, please give us a star! 🙏 \n\n## Showcases\n\n* **[Tick-level backtesting](https://github.com/marsupialtail/quokka/blob/master/blog/backtest.md):** backtest a mid-high frequency trading strategy against SIP trade stream for the last four years in 10 minutes.\n\n* **[Vector embedding analytics](https://blog.lancedb.com/why-dataframe-libraries-need-to-understand-vector-embeddings-291343efd5c8):** easily add new input readers in Python, like for the [Lance](https://github.com/lancedb/lance) format.\n\n* **[Extreme feature engineering on 10k columns](https://github.com/marsupialtail/quokka/blob/master/blog/approxquant.md):** easily integrate with Arrow-compatible C++ Plugins.\n\n* **[TPC-H](https://github.com/marsupialtail/quokka/blob/master/blog/release.md):** Several times faster than SparkSQL in many TPC-H queries. (EMR, not DBR!)\n\n\u003cp align=\"center\"\u003e\n  \u003cimg src=\"https://github.com/marsupialtail/quokka/blob/master/docs/docs/tpch-parquet.svg?raw=true\" alt=\"Title\"/\u003e\n\u003c/p\u003e\n\n* **Detect iceberg orders in quote stream (upcoming):** use complex event processing to easily detect iceberg order in the MBO stream.\n* **Backtest an online learning algorithm on clickstream data (upcoming):** test incremental training algorithms on historical data.\n\n## What is Quokka?\n\nIn technical terms, Quokka is a **push-based distributed query engine with lineage-based fault tolerance**.\n\nIn practical terms, Quokka is a tool for you to run **custom stateful and windowed computation over terabytes of historical time series data**. It supports regular SQL like filter, selection and joins with relational optimizations like predicate pushdown, early selection and join reordering, but truly excels on time series workloads like complicated windows, asof/range joins, complex pattern recognition (SPL transactions, EQL sequence, SQL Match_Recognize) and custom stateful computations like building a limit order book or testing online learning strategies.\n\nUnlike most other query engines, Quokka is implemented completely in Python and is meant to be easily extensible for new operations and use cases, e.g. time series analytics and feature engineering. It is truly simple to extend, because *all* you have to do is raise a Github issue and more likely than not I'll write the operator for you.\n\nQuokka operates on DataStreams, which are basically Spark RDDs except data partitions can be produced serially. A data partition can be consumed immediately after it's produced, unlike Spark where all the partitions have to be present in the RDD before the shuffle happens. This allows Quokka to pipeline multiple shuffles and I/O, leading to large performance gains.\n\n## Quick Start\n\nQuokka requires Redis \u003e 6.2. You can install latest Redis using: \n\n~~~bash\ncurl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\necho \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\nsudo apt-get update\nsudo apt-get install redis\n~~~\n\nMake sure to run `redis-server -v` to check the Redis Server version! The default `apt-get install` version is most likely wrong.\n\n~~~python\npip3 install pyquokka\n~~~\n**Docs**: https://marsupialtail.github.io/quokka/\n\nQuokka offers a DataStream API that resembles Spark's DataFrame API. You can create a DataStream from a Polars Dataframe easily for local testing. \n\n~~~python\n\u003e\u003e\u003e from pyquokka import QuokkaContext\n\u003e\u003e\u003e qc = QuokkaContext()\n\u003e\u003e\u003e import polars\n\u003e\u003e\u003e a = polars.from_dict({\"a\":[1,1,2,2], \"b\":['{\"my_field\": \"quack\"}','{\"my_field\": \"quack\"}','{\"my_field\": \"quack\"}','{\"my_field\": \"quack\"}']})\n\u003e\u003e\u003e a\nshape: (4, 2)\n┌─────┬───────────────────────┐\n│ a   ┆ b                     │\n│ --- ┆ ---                   │\n│ i64 ┆ str                   │\n╞═════╪═══════════════════════╡\n│ 1   ┆ {\"my_field\": \"quack\"} │\n│ 1   ┆ {\"my_field\": \"quack\"} │\n│ 2   ┆ {\"my_field\": \"quack\"} │\n│ 2   ┆ {\"my_field\": \"quack\"} │\n└─────┴───────────────────────┘\n\u003e\u003e\u003e b = qc.from_polars(a)\n\u003e\u003e\u003e b\nDataStream[a,b]\n\n# DataStreams are lazy, you must call collect to get the values, like a Polars LazyFrame or Spark DataFrame.\n\u003e\u003e\u003e b.collect()\nshape: (4, 2)\n┌─────┬───────────────────────┐\n│ a   ┆ b                     │\n│ --- ┆ ---                   │\n│ i64 ┆ str                   │\n╞═════╪═══════════════════════╡\n│ 1   ┆ {\"my_field\": \"quack\"} │\n│ 1   ┆ {\"my_field\": \"quack\"} │\n│ 2   ┆ {\"my_field\": \"quack\"} │\n│ 2   ┆ {\"my_field\": \"quack\"} │\n└─────┴───────────────────────┘\n\n# Quokka supports filtering by a SQL statement directly\n\u003e\u003e\u003e b.filter_sql(\"a==1\").collect()\nshape: (2, 2)\n┌─────┬───────────────────────┐\n│ a   ┆ b                     │\n│ --- ┆ ---                   │\n│ i64 ┆ str                   │\n╞═════╪═══════════════════════╡\n│ 1   ┆ {\"my_field\": \"quack\"} │\n│ 1   ┆ {\"my_field\": \"quack\"} │\n└─────┴───────────────────────┘\n\n~~~\n\nCurrently Quokka supports reading data from CSV/Parquet on disk/S3, Apache Iceberg through AWS Glue and Ray Datasets, though theoretically any data source can be supported: Delta Lake/Hudi, S3 bucket of images, transactional database CDC endpoints etc. If you have some esoteric data source that you want to run analytics on, please send me a challenge as a Github issue, or better yet make a pull request. The following code showcases some more of Quokka's APIs.\n\n~~~python\n\u003e\u003e\u003e lineitem = qc.read_parquet(s3_path_parquet + \"lineitem.parquet/*\")\n\u003e\u003e\u003e orders = qc.read_parquet(s3_path_parquet + \"orders.parquet/*\")\n# filter with dataframe syntax\n\u003e\u003e\u003e d = lineitem.filter(lineitem[\"l_commitdate\"] \u003c lineitem[\"l_receiptdate\"])\n\u003e\u003e\u003e d = orders.join(d, left_on=\"o_orderkey\", right_on=\"l_orderkey\", how = \"semi\")\n# filter with SQL syntax\n\u003e\u003e\u003e d = d.filter_sql(\"o_orderdate \u003e= date '1993-07-01'\")\n\u003e\u003e\u003e d = d.select([\"o_orderkey\", \"o_custkey\", \"o_orderstatus\", \"o_totalprice\", \"o_orderdate\", \"o_orderpriority\", \"o_clerk\", \"o_shippriority\"])\n\u003e\u003e\u003e d.explain() # this will produce a PDF execution graph you can visualize \n\u003e\u003e\u003e d = d.compute()\n\u003e\u003e\u003e dataset = d.to_ray_dataset()\n~~~\n\n## Fineprint\n\nQuokka should not be used as a replacement for SparkSQL (it doesn't parse SQL directly yet, though it is on the roadmap). Instead you can play with it to see if it can give you better performance for your use cases. Another strength of Quokka is that it's Python-native, so you will never have to worry about JVM errors when you start using hairy UDFs with custom Python packages.\n\nQuokka stands on the shoulders of giants. It uses [Ray](https://github.com/ray-project/ray) for task scheduling, Redis for lineage logging, [Polars](https://github.com/pola-rs/polars) and DuckDB for relational algebra kernels and [Apache Arrow](https://github.com/apache/arrow) for I/O. All of those are backed by efficient C++/Rust implementations. \n\nPlease refer to the [docs](https://marsupialtail.github.io/quokka/) and examples in the apps directory. \n\nFor any questions/concerns/just want to chat: zihengw@stanford.edu, or join the [Discord](https://discord.gg/6ujVV9HAg3) channel. Please do reach out before you use Quokka for anything real. Please raise a Github issue if you encounter any issues.\n\nShoutout to Emanuel Adamiak and Sarah Fujimori, who contributed significantly to this project. Also huge thanks to my advisor, Alex Aiken, as well as Tim Tully, Matei Zaharia, Peter Kraft, Qian Li and Toby Mao for all the help throughput the years. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarsupialtail%2Fquokka","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmarsupialtail%2Fquokka","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmarsupialtail%2Fquokka/lists"}