{"id":13877827,"url":"https://github.com/launchflow/buildflow","last_synced_at":"2025-07-16T13:32:48.522Z","repository":{"id":73406463,"uuid":"592379104","full_name":"launchflow/buildflow","owner":"launchflow","description":"BuildFlow, is an open source framework for building large scale systems using Python. All you need to do is describe where your input is coming from and where your output should be written, and BuildFlow handles the rest. No configuration outside of the code is required.","archived":false,"fork":false,"pushed_at":"2024-01-10T21:58:13.000Z","size":3772,"stargazers_count":193,"open_issues_count":38,"forks_count":7,"subscribers_count":3,"default_branch":"main","last_synced_at":"2024-10-06T09:08:10.999Z","etag":null,"topics":["batch","data-science","pipeline","python","streaming"],"latest_commit_sha":null,"homepage":"https://docs.launchflow.com/buildflow","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/launchflow.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":"CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null}},"created_at":"2023-01-23T15:55:35.000Z","updated_at":"2024-07-31T07:45:39.000Z","dependencies_parsed_at":"2023-10-13T05:14:04.447Z","dependency_job_id":"cba0e82f-9638-4f7c-bc27-97337b981e6c","html_url":"https://github.com/launchflow/buildflow","commit_stats":null,"previous_names":[],"tags_count":18,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/launchflow%2Fbuildflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/launchflow%2Fbuildflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/launchflow%2Fbuildflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/launchflow%2Fbuildflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/launchflow","download_url":"https://codeload.github.com/launchflow/buildflow/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":226134226,"owners_count":17578778,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["batch","data-science","pipeline","python","streaming"],"created_at":"2024-08-06T08:01:26.705Z","updated_at":"2024-11-24T06:31:32.557Z","avatar_url":"https://github.com/launchflow.png","language":"Python","funding_links":[],"categories":["Python"],"sub_categories":[],"readme":"\u003cdiv align=\"center\"\u003e\n\n\u003cimg src=\"https://raw.githubusercontent.com/launchflow/buildflow/main/docs/images/buildflow-light.png#gh-light-mode-only\" alt=\"BuildFlow Logo\"\u003e\n\u003cimg src=\"https://raw.githubusercontent.com/launchflow/buildflow/main/docs/images/buildflow-dark.png#gh-dark-mode-only\" alt=\"BuildFlow Logo\"\u003e\n\n\u003chr\u003e\n\n### **⚒️ Build your entire system in minutes using pure Python. ⚒️**\n\n![CI](https://github.com/launchflow/buildflow/actions/workflows/python_ci.yaml/badge.svg)\n![Release Tests](https://github.com/launchflow/buildflow/actions/workflows/release_tests.yaml/badge.svg)\n[![Python version](https://badge.fury.io/py/buildflow.svg)](https://pypi.org/project/buildflow)\n[![codecov](https://codecov.io/gh/launchflow/buildflow/branch/main/graph/badge.svg?token=AO0TP8XG7X)](https://codecov.io/gh/launchflow/buildflow)\n[![Discord](https://dcbadge.vercel.app/api/server/jRpkTAeEWx?style=flat)](https://discord.gg/jRpkTAeEWx)\n\n\n\u003c/div\u003e\n\n## 📑 Resources\n\n\u003cdiv align=\"center\"\u003e\n\n📖 [Docs](https://docs.launchflow.com/buildflow/introduction) \u0026nbsp; | \u0026nbsp; ⚡ [Quickstart](https://docs.launchflow.com/buildflow/quickstart) \u0026nbsp; | \u0026nbsp; 👋 [Slack](https://join.slack.com/t/launchflowusers/shared_invite/zt-27wlowsza-Uiu~8hlCGkvPINjmMiaaMQ) \u0026nbsp; | \u0026nbsp; 🌟 [Contribute](https://docs.launchflow.com/buildflow/developers/contribute) \u0026nbsp; | \u0026nbsp; 🚀 [Deployment](https://www.launchflow.com/) \u0026nbsp;\n\n\u003c/div\u003e\n\n## 🤔 What is BuildFlow?\n\n**BuildFlow** is a Python framework that allows you to build your entire backend system using one framework. With our simple decorator pattern you can turn any function into a component of your backend system. Allowing you to **serve data over HTTP, dump data to a datastore, or process async data from message queues**. All of these can use our built in IO connectors allowing you to create, manage, and connect to your cloud resources using pure Python.\n\n### Key Features\n\n#### Common Serving \u0026 Processing Patterns | 📖 [Docs](https://docs.launchflow.com/buildflow/concepts#processors)\n\nTurn any function into a component of your backend system.\n\n```python\n# Serve traffic over HTTP or Websockets\nservice = app.service(\"my-service\")\n@service.endpoint(\"/\", method=\"GET\")\ndef get():\n    return \"Hello World\"\n\n# Collect, transform, and write data to storage\n@app.collector(\"/collect\", method=\"POST\", sink=SnowflakeTable(...))\ndef collect(request: Dict[str, Any]):\n  return element\n\n# Process data from message queues such as Pub/Sub \u0026 SQS\n@app.consumer(source=SQSQueue(...), sink=BigQuery(...))\ndef process(element: Dict[str, Any]):\n    return element\n```\n\n#### Infrastructure from Code | 📖 [Docs](https://docs.launchflow.com/buildflow/guides/manage-cloud-resources)\n\nCreate and connect to cloud resources using python (powered by [Pulumi](https://github.com/pulumi/pulumi))\n\n```python\n# Use Python objects to define your infrastructure\nsqs_queue = SQSQueue(\"queue-name\")\ngcs_bucket = GCSBucket(\"bucket-name\")\n\n# Your application manages its own infrastructure state\napp.manage(s3_bucket, gcs_bucket)\n\n# Use the same resource objects in your application logic\n@app.consumer(source=sqs_queue, sink=gcs_bucket)\ndef process(event: YourSchema) -\u003e OutputSchema:\n    # Processing logic goes here\n    return OutputSchema(...)\n```\n\n#### Dependency Injection | 📖 [Docs](https://docs.launchflow.com/buildflow/programming-guide/dependencies)\n\nInject any dependency with full control over its setup and lifecycle\n\n```python\n# Define custom dependencies\n@dependency(Scope.GLOBAL)\nclass MyStringDependency:\n    def __init__(self):\n        self.my_string = \"HELLO!\"\n\n# Or use the prebuilt dependencies\nPostgresDep = SessionDepBuilder(postgres)\n\n# BuildFlow handles the rest\n@service.endpoint(\"/hello\", method=\"GET\")\ndef hello(db: PostgresDep, custom_dep: MyStringDependency):\n    with db.session as session:\n        user = session.query(User).first()\n    # Returns \"HELLO! User.name\"\n    return f\"{custom_dep.my_string} {user.name}\"\n```\n\n#### Async Runtime | 📖 [Docs](https://docs.launchflow.com/buildflow/programming-guide/processors#async-with-ray)\n\nScale out parallel tasks across your cluster with [Ray](https://docs.ray.io/en/latest/index.html) or any other async framework.\n\n```python\n@ray.remote\ndef long_task(elem):\n    time.sleep(10)\n    return elem\n\n@app.consumer(PubSubSubscription(...), BigQueryTable(...))\ndef my_consumer(elem):\n    # Tasks are automatically parallelized across your cluster\n    return await long_task.remote(elem)\n```\n\n## ⚙️ Installation\n\n```bash\npip install buildflow\n```\n\n### Extra Dependencies\n\n#### Pulumi Installation\n\nBuildFlow uses Pulumi to manage resources used by your application. To install Pulumi visit: https://www.pulumi.com/docs/install/\n\nInstalling Pulumi unlocks:\n\n- allows BuildFlow to manage resource creation and destruction\n- full access to Pulumi API / CLI\n- fine-grained control over Pulumi Stacks \u0026 Resources\n\n## 🩺 Code Health Checks\n\nWe use [black](https://github.com/psf/black) and [ruff](https://github.com/charliermarsh/ruff) with [pre-commit](https://pre-commit.com/) hooks to perform health checks.\nTo setup these locally:\n\n- Clone the repo\n- Install the `dev` dependencies like `python -m pip install -e .[dev]`\n- Check if pre-commit is installed correctly by running `pre-commit --version`\n- Setup pre-commit to run before every commit on staged changes by running `pre-commit install`\n- Pre-commit can also be ran manually as `pre-commit run --all-files`\n\n## 📜 License\n\nBuildFlow is open-source and licensed under the [Apache License 2.0](LICENSE). We welcome contributions, see our [CONTRIBUTING.md](CONTRIBUTING.md) file for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaunchflow%2Fbuildflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Flaunchflow%2Fbuildflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Flaunchflow%2Fbuildflow/lists"}