Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/launchflow/buildflow
BuildFlow, is an open source framework for building large scale systems using Python. All you need to do is describe where your input is coming from and where your output should be written, and BuildFlow handles the rest. No configuration outside of the code is required.
https://github.com/launchflow/buildflow
batch data-science pipeline python streaming
Last synced: about 2 months ago
JSON representation
BuildFlow, is an open source framework for building large scale systems using Python. All you need to do is describe where your input is coming from and where your output should be written, and BuildFlow handles the rest. No configuration outside of the code is required.
- Host: GitHub
- URL: https://github.com/launchflow/buildflow
- Owner: launchflow
- License: apache-2.0
- Created: 2023-01-23T15:55:35.000Z (over 1 year ago)
- Default Branch: main
- Last Pushed: 2024-01-10T21:58:13.000Z (8 months ago)
- Last Synced: 2024-07-14T17:36:31.758Z (2 months ago)
- Topics: batch, data-science, pipeline, python, streaming
- Language: Python
- Homepage: https://docs.launchflow.com/buildflow
- Size: 3.6 MB
- Stars: 190
- Watchers: 3
- Forks: 7
- Open Issues: 38
-
Metadata Files:
- Readme: README.md
- Changelog: CHANGELOG.md
- Contributing: CONTRIBUTING.md
- License: LICENSE
- Code of conduct: CODE_OF_CONDUCT.md
- Codeowners: CODEOWNERS
Awesome Lists containing this project
README
### **⚒️ Build your entire system in minutes using pure Python. ⚒️**
[](https://pypi.org/project/buildflow)
[](https://codecov.io/gh/launchflow/buildflow)
[](https://discord.gg/jRpkTAeEWx)
## 📑 Resources
📖 [Docs](https://docs.launchflow.com/buildflow/introduction) | ⚡ [Quickstart](https://docs.launchflow.com/buildflow/quickstart) | 👋 [Slack](https://join.slack.com/t/launchflowusers/shared_invite/zt-27wlowsza-Uiu~8hlCGkvPINjmMiaaMQ) | 🌟 [Contribute](https://docs.launchflow.com/buildflow/developers/contribute) | 🚀 [Deployment](https://www.launchflow.com/)
## 🤔 What is BuildFlow?
**BuildFlow** is a Python framework that allows you to build your entire backend system using one framework. With our simple decorator pattern you can turn any function into a component of your backend system. Allowing you to **serve data over HTTP, dump data to a datastore, or process async data from message queues**. All of these can use our built in IO connectors allowing you to create, manage, and connect to your cloud resources using pure Python.
### Key Features
#### Common Serving & Processing Patterns | 📖 [Docs](https://docs.launchflow.com/buildflow/concepts#processors)
Turn any function into a component of your backend system.
```python
# Serve traffic over HTTP or Websockets
service = app.service("my-service")
@service.endpoint("/", method="GET")
def get():
return "Hello World"
# Collect, transform, and write data to storage
@app.collector("/collect", method="POST", sink=SnowflakeTable(...))
def collect(request: Dict[str, Any]):
return element
# Process data from message queues such as Pub/Sub & SQS
@app.consumer(source=SQSQueue(...), sink=BigQuery(...))
def process(element: Dict[str, Any]):
return element
```
#### Infrastructure from Code | 📖 [Docs](https://docs.launchflow.com/buildflow/guides/manage-cloud-resources)
Create and connect to cloud resources using python (powered by [Pulumi](https://github.com/pulumi/pulumi))
```python
# Use Python objects to define your infrastructure
sqs_queue = SQSQueue("queue-name")
gcs_bucket = GCSBucket("bucket-name")
# Your application manages its own infrastructure state
app.manage(s3_bucket, gcs_bucket)
# Use the same resource objects in your application logic
@app.consumer(source=sqs_queue, sink=gcs_bucket)
def process(event: YourSchema) -> OutputSchema:
# Processing logic goes here
return OutputSchema(...)
```
#### Dependency Injection | 📖 [Docs](https://docs.launchflow.com/buildflow/programming-guide/dependencies)
Inject any dependency with full control over its setup and lifecycle
```python
# Define custom dependencies
@dependency(Scope.GLOBAL)
class MyStringDependency:
def __init__(self):
self.my_string = "HELLO!"
# Or use the prebuilt dependencies
PostgresDep = SessionDepBuilder(postgres)
# BuildFlow handles the rest
@service.endpoint("/hello", method="GET")
def hello(db: PostgresDep, custom_dep: MyStringDependency):
with db.session as session:
user = session.query(User).first()
# Returns "HELLO! User.name"
return f"{custom_dep.my_string} {user.name}"
```
#### Async Runtime | 📖 [Docs](https://docs.launchflow.com/buildflow/programming-guide/processors#async-with-ray)
Scale out parallel tasks across your cluster with [Ray](https://docs.ray.io/en/latest/index.html) or any other async framework.
```python
@ray.remote
def long_task(elem):
time.sleep(10)
return elem
@app.consumer(PubSubSubscription(...), BigQueryTable(...))
def my_consumer(elem):
# Tasks are automatically parallelized across your cluster
return await long_task.remote(elem)
```
## ⚙️ Installation
```bash
pip install buildflow
```
### Extra Dependencies
#### Pulumi Installation
BuildFlow uses Pulumi to manage resources used by your application. To install Pulumi visit: https://www.pulumi.com/docs/install/
Installing Pulumi unlocks:
- allows BuildFlow to manage resource creation and destruction
- full access to Pulumi API / CLI
- fine-grained control over Pulumi Stacks & Resources
## 🩺 Code Health Checks
We use [black](https://github.com/psf/black) and [ruff](https://github.com/charliermarsh/ruff) with [pre-commit](https://pre-commit.com/) hooks to perform health checks.
To setup these locally:
- Clone the repo
- Install the `dev` dependencies like `python -m pip install -e .[dev]`
- Check if pre-commit is installed correctly by running `pre-commit --version`
- Setup pre-commit to run before every commit on staged changes by running `pre-commit install`
- Pre-commit can also be ran manually as `pre-commit run --all-files`
## 📜 License
BuildFlow is open-source and licensed under the [Apache License 2.0](LICENSE). We welcome contributions, see our [CONTRIBUTING.md](CONTRIBUTING.md) file for details.