{"id":13626086,"url":"https://github.com/feldera/feldera","last_synced_at":"2026-04-13T10:01:30.479Z","repository":{"id":166531879,"uuid":"639599253","full_name":"feldera/feldera","owner":"feldera","description":"The Feldera Incremental Computation Engine","archived":false,"fork":false,"pushed_at":"2026-04-09T04:46:53.000Z","size":221578,"stargazers_count":1858,"open_issues_count":533,"forks_count":109,"subscribers_count":10,"default_branch":"main","last_synced_at":"2026-04-09T05:16:27.979Z","etag":null,"topics":["data-analytics","data-pipelines","database","incremental-computation","incremental-view-maintenance","ivm","materialized-views","real-time","rust","sql","streaming"],"latest_commit_sha":null,"homepage":"https://feldera.com","language":"Rust","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/feldera.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2023-05-11T19:57:19.000Z","updated_at":"2026-04-09T04:46:55.000Z","dependencies_parsed_at":"2026-04-01T21:02:11.275Z","dependency_job_id":"790a0e90-ec7f-4776-a756-bd75886a5f5e","html_url":"https://github.com/feldera/feldera","commit_stats":null,"previous_names":["feldera/dbsp","feldera/feldera"],"tags_count":284,"template":false,"template_full_name":null,"purl":"pkg:github/feldera/feldera","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feldera%2Ffeldera","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feldera%2Ffeldera/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feldera%2Ffeldera/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feldera%2Ffeldera/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/feldera","download_url":"https://codeload.github.com/feldera/feldera/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/feldera%2Ffeldera/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31747177,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-13T09:16:15.125Z","status":"ssl_error","status_checked_at":"2026-04-13T09:16:05.023Z","response_time":93,"last_error":"SSL_connect returned=1 errno=0 peeraddr=140.82.121.6:443 state=error: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-analytics","data-pipelines","database","incremental-computation","incremental-view-maintenance","ivm","materialized-views","real-time","rust","sql","streaming"],"created_at":"2024-08-01T21:02:09.639Z","updated_at":"2026-04-13T10:01:30.446Z","avatar_url":"https://github.com/feldera.png","language":"Rust","readme":"\u003ch1 align=\"center\"\u003e\n  \u003ca href=\"https://feldera.com\"\u003e\n    \u003cpicture\u003e\n      \u003csource height=\"125\" media=\"(prefers-color-scheme: dark)\" srcset=\"https://raw.githubusercontent.com/feldera/docs.feldera.com/refs/heads/main/img/logo-color-light.svg\"\u003e\n      \u003cimg height=\"125\" alt=\"Feldera\" src=\"https://raw.githubusercontent.com/feldera/docs.feldera.com/refs/heads/main/img/logo.svg\"\u003e\n    \u003c/picture\u003e\n  \u003c/a\u003e\n  \u003cbr\u003e\n  \u003cbr\u003e\n  \u003ca href=\"https://opensource.org/licenses/MIT\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/License-MIT-green.svg\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://github.com/feldera/feldera/actions/workflows/ci.yml\"\u003e\n    \u003cimg src=\"https://github.com/feldera/feldera/actions/workflows/ci.yml/badge.svg?event=merge_group\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://www.feldera.com/community\"\u003e\n    \u003cimg salt=\"Slack\" src=\"https://img.shields.io/badge/slack-blue.svg?logo=slack\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://discord.gg/5YBX9Uw5u7\"\u003e\n    \u003cimg alt=\"Discord\" src=\"https://img.shields.io/badge/discord-blue.svg?logo=discord\u0026logoColor=white\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://try.feldera.com/\"\u003e\n    \u003cimg alt=\"Sandbox\" src=\"https://img.shields.io/badge/feldera_sandbox-blue?logo=CodeSandbox\"\u003e\n  \u003c/a\u003e\n  \u003ca href=\"https://crates.io/crates/dbsp\"\u003e\n    \u003cimg alt=\"crates.io\" src=\"https://img.shields.io/crates/v/dbsp.svg\"\u003e\n  \u003c/a\u003e\n\u003c/h1\u003e\n\n\u003cp align=\"center\"\u003e\n  \u003cem\u003e\u003cb\u003e\u003ca href=\"https://feldera.com\"\u003eFeldera\u003c/a\u003e\u003c/b\u003e\u003c/em\u003e is a fast query engine for \u003cb\u003eincremental computation\u003c/b\u003e. Feldera has the \u003ca href=\"#-theory\"\u003eunique\u003c/a\u003e ability to \u003cb\u003eevaluate arbitrary SQL programs incrementally\u003c/b\u003e, making it more powerful, expressive and performant than existing alternatives like batch engines, warehouses, stream processors or streaming databases.\n\u003c/p\u003e\n\n---\n\n## 🔥 Incremental Computation Engine\n\nOur approach to incremental computation is simple. A Feldera `pipeline` is a set of SQL tables and views. Views can be\ndeeply nested.\nUsers start, stop or pause pipelines to manage and advance a computation.\nPipelines continuously process\n**changes**, which are any number of inserts, updates or deletes to a set of tables. When the pipeline receives changes,\nFeldera **incrementally** updates all the views by only looking at the changes and it completely avoids recomputing over\nolder data.\nWhile a pipeline is running, users can inspect the results of the views at any time.\n\nOur approach to incremental computation makes Feldera incredibly fast (millions of events per second on a laptop).\nIt also enables **unified offline and online compute** over both live and historical data. Feldera users have built batch\nand real-time\nfeature engineering pipelines, ETL pipelines, various forms of incremental and periodic analytical jobs over batch data,\nand more.\n\n## 🎯 Our defining Features\n\n1. **Full SQL support and more.**  Our engine is the only one in existence that can evaluate full SQL\n   syntax and semantics completely incrementally. This includes joins and aggregates, group by, correlated subqueries,\n   window functions, complex data types, time series operators, UDFs, and\n   recursive queries. Pipelines can process deeply nested hierarchies of views.\n\n2. **Fast out-of-the-box performance.**  Feldera users have reported getting complex use cases\n   implemented in 30 minutes or less, and hitting millions\n   of events per second in performance on a laptop without any tuning.\n\n3. **Datasets larger than RAM.** Feldera is designed to handle datasets\n   that exceed the available RAM by spilling efficiently to disk, taking advantage of recent advances in NVMe storage.\n\n4. **Strong guarantees on consistency and freshness.** Feldera is strongly consistent. It\n   also [guarantees](https://www.feldera.com/blog/synchronous-streaming/) that the state of the views always corresponds\n   to what you'd get if you ran the queries in a batch system for the same input.\n\n5. **Connectors for your favorite data sources and destinations.** Feldera connects to myriad batch and streaming data\n   sources, like Kafka, HTTP, CDC streams, S3, Data Lakes, Warehouses and more.\n   If you need a connector that we don't yet support, [let us know](https://github.com/feldera/feldera/issues).\n\n6. **Fault tolerance**. Feldera can gracefully restart from the exact\n   point of an abrupt shutdown or crash, picking up from where it left\n   off without dropping or duplicating input or output. Fault\n   tolerance is a preview feature that requires support from input and\n   output connectors.\n\n7. **Seamless ad-hoc queries**. You can run ad-hoc SQL queries on a running or paused pipeline to inspect or debug the\n   state of materialized views. While these queries are evaluated in batch mode using Apache Datafusion, their\n   results are consistent with the incremental engine's output for the same queries, aside from minor dialect and\n   rounding differences.\n\n## 💻 Architecture\n\nThe following diagram shows Feldera's architecture\n\n![Feldera Platform Architecture](architecture.svg)\n\n## ⚡️ Quick start with Docker\n\nFirst, make sure you have [Docker](https://docs.docker.com/) installed. Then run the\nfollowing command:\n\n```text\ndocker run -p 8080:8080 --tty --rm -it images.feldera.com/feldera/pipeline-manager:latest\n```\n\nOnce the container image downloads and you see the Feldera logo on your terminal, visit\nthe WebConsole at [http://localhost:8080](http://localhost:8080).\nWe suggest going through our [tutorial](https://docs.feldera.com/tutorials/basics/) next.\n\nWe also have instructions to run Feldera using [Docker Compose](https://docs.feldera.com/get-started),\nif you'd like to experiment with Kafka and other auxiliary services.\n\n## ⚙️ Running Feldera from sources\n\nTo run Feldera from sources, ensure at least 6 GB of free space in the sources directory and an additional 7 GB in your home directory, then install the required dependencies:\n\n- [Rust tool chain](https://www.rust-lang.org/tools/install)\n- cmake\n- libssl-dev\n- libsasl2-dev\n- golang-go (required to build aws-lc-fips-sys when using rustls FIPS)\n- pkg-config\n- libzstd-dev\n- clang\n- Java Development Kit (JDK), version 19 or newer (21 is recommended)\n- maven\n- [Bun](https://bun.sh/docs/installation)\n- [nodejs v20](https://github.com/nodesource/distributions/blob/master/DEV_README.md)\n\nAfter that, the first step is to build the SQL compiler:\n\n```\ncd sql-to-dbsp-compiler\n./build.sh\n```\n\nNext, from the repository root, run the pipeline-manager:\n\n```\ncargo run --bin=pipeline-manager\n```\n\nAs with the Docker instructions above, you can now visit\n[http://localhost:8080](http://localhost:8080) on your browser to see the\nFeldera WebConsole.\n\n## 📖 Documentation\n\nTo learn more about Feldera Platform, we recommend going through the\n[documentation](https://docs.feldera.com).\n\n* [Getting started](https://docs.feldera.com/get-started)\n* [Feldera basics](https://docs.feldera.com/tutorials/basics/)\n* [Tutorials](https://docs.feldera.com/tutorials)\n* [SQL reference](https://docs.feldera.com/sql/)\n* [API reference](https://docs.feldera.com/api)\n* [Python SDK](https://docs.feldera.com/python/)\n\n## 🤖 Benchmarks\n\nFeldera is generally [faster and uses less memory](https://www.feldera.com/blog/nexmark-vs-flink)\nthan systems like stream processors.\n\n\u003cp float=\"left\" align=\"middle\"\u003e\n  \u003cimg src=\"https://www.feldera.com/_next/image?url=https://cdn.sanity.io/images/nlte859i/production/c80a9d592fb6f6e4cf2c7a665add24da65998123-1740x493.png?D75\u0026fit=clip\u0026auto=format\u0026w=1920\u0026q=100\" width=\"100%\"\u003e\n\u003c/p\u003e\n\n## 👍 Contributing\n\nThe software in this repository is governed by an open-source license.\nWe welcome contributions. Here are some [guidelines](CONTRIBUTING.md).\n\n## 🎓 Theory\n\nFeldera Platform achieves its objectives by building on a solid mathematical\nfoundation. The formal model that underpins our system, called DBSP, is\ndescribed in the accompanying paper:\n\n- [Budiu, Chajed, McSherry, Ryzhyk, Tannen. DBSP: Automatic\n  Incremental View Maintenance for Rich Query Languages, Conference on\n  Very Large Databases, August 2023, Vancouver,\n  Canada](https://docs.feldera.com/vldb23.pdf)\n\n- Here is [a presentation about DBSP](https://www.youtube.com/watch?v=iT4k5DCnvPU) at the 2023\n  Apache Calcite Meetup.\n\nThe model provides two things:\n\n1. **Semantics.** DBSP defines a formal language of streaming operators and\n   queries built out of these operators, and precisely specifies how these queries\n   must transform input streams to output streams.\n\n1. **Algorithm.** DBSP also gives an algorithm that takes an arbitrary query and\n   generates an incremental dataflow program that implements this query correctly (in accordance\n   with its formal semantics) and efficiently. Efficiency here means, in a\n   nutshell, that the cost of processing a set of input events is proportional to\n   the size of the input rather than the entire state of the database.\n\n","funding_links":[],"categories":["Rust","streaming"],"sub_categories":[],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeldera%2Ffeldera","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Ffeldera%2Ffeldera","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Ffeldera%2Ffeldera/lists"}