{"id":13688768,"url":"https://github.com/bytewax/bytewax","last_synced_at":"2025-05-13T18:14:39.440Z","repository":{"id":37839800,"uuid":"455655192","full_name":"bytewax/bytewax","owner":"bytewax","description":"Python Stream Processing","archived":false,"fork":false,"pushed_at":"2025-03-27T13:25:59.000Z","size":12531,"stargazers_count":1726,"open_issues_count":33,"forks_count":79,"subscribers_count":18,"default_branch":"main","last_synced_at":"2025-05-08T05:44:18.789Z","etag":null,"topics":["data-engineering","data-processing","data-science","dataflow","machine-learning","python","rust","stream-processing","streaming-data"],"latest_commit_sha":null,"homepage":"https://docs.bytewax.io/","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/bytewax.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2022-02-04T18:29:36.000Z","updated_at":"2025-05-07T17:19:27.000Z","dependencies_parsed_at":"2023-11-13T11:25:54.369Z","dependency_job_id":"8b0f144a-7f11-4f04-a312-e42fa367ca6a","html_url":"https://github.com/bytewax/bytewax","commit_stats":{"total_commits":2059,"total_committers":27,"mean_commits":76.25925925925925,"dds":0.6823700825643516,"last_synced_commit":"c512ffe88441ebd6f751f5f6724e2c66952e55ac"},"previous_names":[],"tags_count":35,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytewax%2Fbytewax","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytewax%2Fbytewax/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytewax%2Fbytewax/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/bytewax%2Fbytewax/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/bytewax","download_url":"https://codeload.github.com/bytewax/bytewax/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254000885,"owners_count":21997443,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["data-engineering","data-processing","data-science","dataflow","machine-learning","python","rust","stream-processing","streaming-data"],"created_at":"2024-08-02T15:01:22.189Z","updated_at":"2025-05-13T18:14:39.415Z","avatar_url":"https://github.com/bytewax.png","language":"Python","funding_links":[],"categories":["Python","Table of Contents","Data Stream Processing","Projects","\u003ca name=\"Python\"\u003e\u003c/a\u003ePython","Stream Processing"],"sub_categories":["Streaming Engine","Stream Processing","Python Libraries"],"readme":"# Bytewax: Python Stateful Stream Processing Framework\n\n\u003cdiv align='center'\u003e\n \u003ca href=\"https://bytewax.io/\"\u003e\n\u003cpicture\u003e\n  \u003csource media=\"(prefers-color-scheme: dark)\" srcset=\"https://github.com/bytewax/bytewax/assets/53014647/cd47293b-72c9-423c-b010-2c4990206c60\" width=\"350\"\u003e\n  \u003csource media=\"(prefers-color-scheme: light)\" srcset=\"https://github.com/bytewax/bytewax/assets/53014647/f376c9e8-5bd4-4563-ba40-3df8761b13fc\" width=\"350\"\u003e\n  \u003cimg alt=\"Bytewax\" src=\"https://github.com/bytewax/bytewax/assets/53014647/f376c9e8-5bd4-4563-ba40-3df8761b13fc\" width=\"350\"\u003e\n\u003c/picture\u003e\n\u003c/a\u003e\n\u003c/div\u003e\n\n\u003cp align='center'\u003e\n\u003ca href=\"https://github.com/bytewax/bytewax/actions\"\u003e\u003cimg src=\"https://github.com/bytewax/bytewax/workflows/CI/badge.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://pypi.org/project/bytewax/\"\u003e\u003cimg src=\"https://img.shields.io/pypi/v/bytewax.svg\"\u003e\u003c/a\u003e\n\u003ca href=\"https://docs.bytewax.io/stable/guide/index.html\"\u003e\u003cimg src=\"https://img.shields.io/badge/user-guide-brightgreen\"\u003e\u003c/a\u003e\n\u003ca href=\"https://github.com/bytewax/bytewax/blob/main/LICENSE\"\u003e\u003cimg src=\"https://img.shields.io/badge/license-Apache--2.0-blue.svg\"\u003e\u003c/a\u003e\n\u003c/p\u003e\n\n\n**Bytewax** is a Python framework and Rust-based distributed processing engine for stateful event and stream processing. Inspired by capabilities found in tools like Apache Flink, Spark, and Kafka Streams, Bytewax makes stream processing simpler and more accessible by integrating directly with the Python ecosystem you already know and trust.\n\n**Key Features:**\n- **Python-first:** Leverage your existing Python libraries, frameworks, and tooling.\n- **Stateful Stream Processing:** Maintain and recover state automatically, enabling advanced online machine learning and complex event-driven applications.\n- **Scalable \u0026 Distributed:** Easily scale from local development to multi-node, multi-worker deployments on Kubernetes or other infrastructures.\n- **Rich Connector Ecosystem:** Ingest data from sources like Kafka, filesystems, or WebSockets, and output to data lakes, key-value stores, or other systems.\n- **Flexible Dataflow API:** Compose pipelines using operators (e.g., `map`, `filter`, `join`, `fold_window`) to express complex logic.\n\n---\n\n## Table of Contents\n\n- [Quick Start](#quick-start)\n- [How Bytewax Works](#how-bytewax-works)\n- [Operators Overview](#operators-overview)\n- [Connectors (Module Hub)](#connectors-module-hub)\n- [Local Development, Testing, and Production](#local-development-testing-and-production)\n- [Deployment Options](#deployment-options)\n  - [Running Locally](#running-locally)\n  - [Containerized Execution](#containerized-execution)\n  - [Scaling on Kubernetes](#scaling-on-kubernetes)\n- [Examples](#examples)\n- [Community and Contributing](#community-and-contributing)\n- [License](#license)\n\n---\n\n## Quick Start\n\nInstall Bytewax from PyPI:\n\n```sh\npip install bytewax\n```\n\n[Install `waxctl`](https://bytewax.io/waxctl) to manage deployments at scale.\n\n**Minimal Example:**\n\n```python\nfrom bytewax.dataflow import Dataflow\nfrom bytewax import operators as op\nfrom bytewax.testing import TestingSource\n\nflow = Dataflow(\"quickstart\")\n\n# Input: Local test source for demonstration\ninp = op.input(\"inp\", flow, TestingSource([1, 2, 3, 4, 5]))\n\n# Transform: Filter even numbers and multiply by 10\nfiltered = op.filter(\"keep_even\", inp, lambda x: x % 2 == 0)\nresults = op.map(\"multiply_by_10\", filtered, lambda x: x * 10)\n\n# Output: Print results to stdout\nop.inspect(\"print_results\", results)\n```\n\nRun it locally:\n\n```sh\npython -m bytewax.run quickstart.py\n```\n\n---\n\n## How Bytewax Works\n\nBytewax uses a **dataflow computational model**, similar to systems like Flink or Spark, but with a Pythonic interface. You define a dataflow graph of operators and connectors:\n\n1. **Input:** Data sources (Kafka, file systems, S3, WebSockets, custom connectors)\n2. **Operators:** Stateful transformations (map, filter, fold_window, join) defined in Python.\n3. **Output:** Data sinks (databases, storage systems, message queues).\n\n\u003cimg width=\"1303\" alt=\"Bytewax Dataflow Animation\" src=\"https://github.com/bytewax/bytewax/assets/156834296/4e314f17-38ab-4e72-9268-a48ddee7a201\"\u003e\n\n**Stateful operations:** Bytewax maintains distributed state, allows for fault tolerance and state recovery, and supports event-time windowing for advanced analytics and machine learning workloads.\n\n**`waxctl`:** Bytewax’s CLI tool for deploying and managing dataflows on cloud servers or Kubernetes clusters. [Download `waxctl` here](https://bytewax.io/waxctl).\n\n---\n\n## Operators Overview\n\nOperators are the building blocks of Bytewax dataflows:\n\n- **Stateless Operators:** `map`, `filter`, `inspect`\n- **Stateful Operators:** `reduce`, `fold_window`, `stateful_map`\n- **Windowing \u0026 Aggregations:** Event-time, processing-time windows, tumbling, sliding, and session windows.\n- **Joins \u0026 Merges:** Combine multiple input streams with `merge`, `join`, or advanced join patterns.\n- **Premium Operators:**\n\nFor a comprehensive list, see the [Operators API Documentation](https://docs.bytewax.io/stable/api/bytewax/bytewax.operators.html).\n\n---\n\n## Connectors\n\nBytewax provides built-in connectors for common data sources and sinks such as Kafka, files, and stdout. You can also write your own [custom connectors](https://docs.bytewax.io/stable/guide/advanced-concepts/custom-connectors.html).\n\n**Examples of Built-in Connectors:**\n- **Kafka:** `bytewax.connectors.kafka`\n- **StdIn/StdOut:** `bytewax.connectors.stdio`\n- **Redis, S3, and More:** See [Bytewax connectors](https://docs.bytewax.io/stable/api/bytewax/bytewax.connectors.html).\n\n**Community \u0026 Partner Connectors:** Check out the [Bytewax Module Hub](https://bytewax.io/modules) for additional connectors contributed by the community.\n\n---\n\n## Local Development, Testing, and Production\n\n**Local Development:**\n- Use `TestingSource` and `inspect` operators for debugging.\n- Iterate quickly by running your flow with `python -m bytewax.run my_flow.py`.\n- Develop custom connectors and sinks locally with Python tooling you already know.\n\n**Testing:**\n- Integration tests: Use `TestingSource` and run flows directly in CI environments.\n- Unit tests: Test individual functions and operators as normal Python code.\n- [More on Testing](https://docs.bytewax.io/stable/guide/contributing/testing.html)\n\n**Production:**\n- Scale horizontally by running multiple workers on multiple machines.\n- Integrate with Kubernetes for dynamic scaling, monitoring, and resilience.\n- Utilize `waxctl` for standardized deployments and lifecycle management.\n\n---\n\n## Deployment Options\n\n### Running Locally\n\nFor experimentation and small-scale jobs:\n\n```sh\npython -m bytewax.run my_dataflow.py\n```\n\nMultiple workers and threads:\n\n```sh\npython -m bytewax.run my_dataflow.py -w 2\n```\n\n### Containerized Execution\n\nRun Bytewax inside Docker containers for easy integration with container platforms. See the [Bytewax Container Guide](https://docs.bytewax.io/stable/guide/deployment/container.html).\n\n### Scaling on Kubernetes\n\nUse `waxctl` to package and deploy Bytewax dataflows to Kubernetes clusters for production workloads:\n\n```sh\nwaxctl df deploy my_dataflow.py --name my-dataflow\n```\n\n[Learn more about Kubernetes deployment](https://docs.bytewax.io/stable/guide/deployment/waxctl.html).\n\n### Scaling with the Bytewax Platform\n\nOur commerically licensed Platform\n\n---\n\n## Examples\n\n- [User Guide](https://docs.bytewax.io/stable/guide/index.html): End-to-end tutorials and advanced topics.\n- [`/examples` Folder](examples): Additional sample dataflows and connector usage.\n\n---\n\n## Community and Contributing\n\nJoin us on [Slack](https://join.slack.com/t/bytewaxcommunity/shared_invite/zt-vkos2f6r-_SeT9pF2~n9ArOaeI3ND2w) for support and discussion.\n\nOpen issues on [GitHub Issues](https://github.com/bytewax/bytewax/issues) for bug reports and feature requests. (For general help, use Slack.)\n\n**Contributions Welcome:**\n- Check out the [Contribution Guide](https://docs.bytewax.io/stable/guide/contributing/contributing.html) to learn how to get started.\n- We follow a [Code of Conduct](https://github.com/bytewax/bytewax/blob/main/CODE_OF_CONDUCT.md).\n\n---\n\n## License\n\nBytewax is licensed under the [Apache-2.0](https://opensource.org/licenses/APACHE-2.0) license.\n\n---\n\n\u003cp align=\"center\"\u003e\n  Built with ❤️ by the Bytewax community\n\u003c/p\u003e\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytewax%2Fbytewax","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbytewax%2Fbytewax","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbytewax%2Fbytewax/lists"}