{"id":20397923,"url":"https://github.com/bcg-x-official/fluxus","last_synced_at":"2025-09-11T21:41:48.372Z","repository":{"id":245125654,"uuid":"812678238","full_name":"BCG-X-Official/fluxus","owner":"BCG-X-Official","description":"Python framework for concurrent data flows","archived":false,"fork":false,"pushed_at":"2024-07-30T00:02:24.000Z","size":2015,"stargazers_count":4,"open_issues_count":2,"forks_count":1,"subscribers_count":4,"default_branch":"1.0.x","last_synced_at":"2025-04-12T13:10:05.924Z","etag":null,"topics":["async","concurrent-programming","data-stream","flow","pipeline","python"],"latest_commit_sha":null,"homepage":"https://bcg-x-official.github.io/fluxus/_generated/home.html","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/BCG-X-Official.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2024-06-09T15:17:24.000Z","updated_at":"2024-10-26T08:44:48.000Z","dependencies_parsed_at":"2025-04-12T13:20:13.305Z","dependency_job_id":null,"html_url":"https://github.com/BCG-X-Official/fluxus","commit_stats":null,"previous_names":["bcg-x-official/fluxus"],"tags_count":5,"template":false,"template_full_name":null,"purl":"pkg:github/BCG-X-Official/fluxus","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BCG-X-Official%2Ffluxus","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BCG-X-Official%2Ffluxus/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BCG-X-Official%2Ffluxus/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BCG-X-Official%2Ffluxus/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/BCG-X-Official","download_url":"https://codeload.github.com/BCG-X-Official/fluxus/tar.gz/refs/heads/1.0.x","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/BCG-X-Official%2Ffluxus/sbom","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":261391996,"owners_count":23151707,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["async","concurrent-programming","data-stream","flow","pipeline","python"],"created_at":"2024-11-15T04:17:25.948Z","updated_at":"2025-06-23T01:05:16.408Z","avatar_url":"https://github.com/BCG-X-Official.png","language":"Python","funding_links":[],"categories":[],"sub_categories":[],"readme":".. image:: sphinx/source/_static/bcgx_logo.png\n   :alt: BCG X logo\n   :width: 200px\n\nIntroduction to *fluxus*\n========================\n\n*fluxus* is a Python framework designed by `BCG X \u003chttps://www.bcg.com/x\u003e`_ to\nstreamline the development of complex data processing pipelines (called *flows*),\nenabling users to quickly and efficiently build, test, and deploy highly concurrent\nworkflows, making complex operations more manageable.\n\nIt is inspired by the data stream paradigm and is designed to be simple,\nexpressive, and composable.\n\nIntroducing Flows\n-----------------\n\nA flow in *fluxus* represents a Directed Acyclic Graph (DAG) where each node performs\na specific operation on the data. These nodes, called *conduits*, are the building\nblocks of a flow, and the data elements that move through the flow are referred to as\n*products*. The conduits are connected to ensure that *products* are processed and\ntransferred correctly from one stage to another.\n\nWithin a *fluxus* flow, there are three main types of conduits:\n\n- **Producers**: These conduits generate or gather raw data from various sources such as\n  databases, APIs, or sensors. They are the entry points of the flow, feeding initial\n  *products* into the system.\n- **Transformers**: These conduits take the *products* from producers and transform\n  them. This can involve filtering, aggregating, enriching, or changing the data to fit\n  the required output or format.\n- **Consumers**: Consumers represent the endpoints of the flow. Each flow has exactly\n  one consumer, which handles the final processed *products*. The consumer may store the\n  data, display it in a user interface, or send it to another system.\n\n\nA Simple Example\n----------------\n\nConsider a simple flow that takes a greeting message, converts it to different cases\n(uppercase, lowercase), and then annotates each message with the case change that\nhas been applied. The flow looks like this:\n\n.. image:: sphinx/source/_images/flow-hello-world.svg\n   :alt: \"Hello World\" flow diagram\n   :width: 600px\n\n\nWith *fluxus*, we can define this flow as follows:\n\n.. code-block:: python\n\n    from fluxus.functional import step, passthrough, run\n\n    input_data = [\n        dict(greeting=\"Hello, World!\"),\n        dict(greeting=\"Bonjour!\"),\n    ]\n\n    def lower(greeting: str):\n        # Convert the greeting to lowercase and keep track of the case change\n        yield dict(\n            greeting=greeting.lower(),\n            case=\"lower\",\n        )\n\n    def upper(greeting: str):\n        # Convert the greeting to uppercase and keep track of the case change\n        yield dict(\n            greeting=greeting.upper(),\n            case=\"upper\",\n        )\n\n    def annotate(greeting: str, case: str = \"original\"):\n        # Annotate the greeting with the case change; default to \"original\"\n        yield dict(greeting=f\"{greeting!r} ({case})\")\n\n    flow = (\n        step(\"input\", input_data)  # initial producer step\n        \u003e\u003e ( # 3 parallel steps: upper, lower, and passthrough\n            step(\"lower\", lower)\n            \u0026 step(\"upper\", upper)\n            \u0026 passthrough()  # passthrough the original input data\n        )\n        \u003e\u003e step(\"annotate\", annotate) # annotate all outputs\n    )\n\n    # Draw the flow diagram\n    flow.draw()\n\nNote the ``passthrough()`` step in the flow. This step is a special type of conduit that\nsimply passes the input data along without modification. This is useful when you want to\nrun multiple transformations in parallel but still want to preserve the original data\nfor further processing.\n\nYou may have noted that the above code does not define a final consumer step. This is\nbecause the ``run`` function automatically adds a consumer step to the end of the flow\nto collect the final output. Custom consumers come into play when you start building\nmore customised flows using the object-oriented API instead of the simpler functional\nAPI we are using here.\n\nWe run the flow with\n\n.. code-block:: python\n\n    result = run(flow)\n\nThis gives us the following output in :code:`result`:\n\n.. code-block:: python\n\n    RunResult(\n        [\n            {\n                'input': {'greeting': 'Hello, World!'},\n                'lower': {'greeting': 'hello, world!', 'case': 'lower'},\n                'annotate': {'greeting': \"'hello, world!' (lower)\"}\n            },\n            {\n                'input': {'greeting': 'Bonjour!'},\n                'lower': {'greeting': 'bonjour!', 'case': 'lower'},\n                'annotate': {'greeting': \"'bonjour!' (lower)\"}\n            }\n        ],\n        [\n            {\n                'input': {'greeting': 'Hello, World!'},\n                'upper': {'greeting': 'HELLO, WORLD!', 'case': 'upper'},\n                'annotate': {'greeting': \"'HELLO, WORLD!' (original)\"}\n            },\n            {\n                'input': {'greeting': 'Bonjour!'},\n                'upper': {'greeting': 'BONJOUR!', 'case': 'upper'},\n                'annotate': {'greeting': \"'BONJOUR!' (original)\"}\n            }\n        ],\n        [\n            {\n                'input': {'greeting': 'Hello, World!'},\n                'annotate': {'greeting': \"'Hello, World!' (original)\"}\n            },\n            {\n                'input': {'greeting': 'Bonjour!'},\n                'annotate': {'greeting': \"'Bonjour!' (original)\"}\n            }\n        ]\n    )\n\nOr, as a *pandas* data frame by calling :code:`result.to_frame()`:\n\n.. image:: sphinx/source/_images/flow-hello-world-results.png\n    :alt: \"Hello World\" flow results\n    :width: 600px\n\nHere's what happened: The flow starts with a single input data item, which is then\npassed along three parallel paths. Each path applies different transformations to the\ndata. The flow then combines the results of these transformations into a single output,\nthe :code:`RunResult`.\n\nNote that the result contains six outputs—one for each of the two input data items along\neach of the three paths through the flow. Also note that the results are grouped as\nseparate lists for each path.\n\nThe run result not only gives us the final product of the ``annotate`` step but also the\ninputs and intermediate products of the ``lower`` and ``upper`` steps. We refer to this\nextended view of the flow results as the *lineage* of the flow.\n\nFor a more thorough introduction to FLUXUS, please visit our\n`User Guide \u003chttps://bcg-x-official.github.io/fluxus/user_guide/index.html\u003e`_.\n\n\nWhy *fluxus*?\n-------------\n\nThe complexity of data processing tasks demands tools that streamline operations and\nensure efficiency. *fluxus* addresses these needs by offering a structured approach to\ncreating flows that handle various data sources and processing requirements. Key\nmotivations for using *fluxus* include:\n\n- **Organisation and Structure**: *fluxus* offers a clear, structured approach to data\n  processing, breaking down complex operations into manageable steps.\n- **Maintainability**: Its modular design allows individual components to be developed,\n  tested, and debugged independently, simplifying maintenance and updates.\n- **Reusability**: Components in *fluxus* can be reused across different projects,\n  reducing development time and effort.\n- **Efficiency**: By supporting concurrent processing, *fluxus* ensures optimal use of\n  system resources, speeding up data processing tasks.\n- **Ease of Use**: *fluxus* provides a functional API that abstracts away the\n  complexities of data processing, making it accessible to developers of all levels.\n  More experienced users can also leverage the advanced features of its underlying\n  object-oriented implementation for additional customisation and versatility (see\n  `User Guide \u003chttps://bcg-x-official.github.io/fluxus/user_guide/index.html\u003e`_ for more\n  details).\n\nConcurrent Processing in *fluxus*\n---------------------------------\n\nA standout feature of *fluxus* is its support for concurrent processing, allowing\nmultiple operations to run simultaneously. This is essential for:\n\n- **Performance**: Significantly reducing data processing time by executing multiple\n  data streams or tasks in parallel.\n- **Resource Utilisation**: Maximising the use of system resources by distributing the\n  processing load across multiple processes or threads.\n\n*fluxus* leverages Python techniques such as threading and asynchronous programming to\nachieve concurrent processing.\n\nBy harnessing the capabilities of *fluxus*, developers can build efficient, scalable,\nand maintainable data processing systems that meet the demands of contemporary\napplications.\n\nGetting started\n===============\n\n- See the\n  `FLUXUS Documentation \u003chttps://bcg-x-official.github.io/fluxus/_generated/home.html\u003e`_\n  for a comprehensive User Guide, API reference, and more.\n- See `Contributing \u003cCONTRIBUTING.md\u003e`_ or visit our detailed\n  `Contributor Guide \u003chttps://bcg-x-official.github.io/fluxus/contributor_guide/index.html\u003e`_\n  for information on contributing.\n- We have an `FAQ \u003chttps://bcg-x-official.github.io/fluxus/faq.html\u003e`_ for common\n  questions. For anything else, please reach out to\n  `artkit@bcg.com \u003cmailto:artkit@bcg.com\u003e`_.\n\n\nUser Installation\n-----------------\n\nInstall using ``pip``:\n\n.. code-block:: bash\n\n    pip install fluxus\n\nor ``conda``:\n\n.. code-block:: bash\n\n    conda install -c bcgx fluxus\n\n\nOptional dependencies\n^^^^^^^^^^^^^^^^^^^^^\n\nTo enable visualizations of flow diagrams, install `GraphViz \u003chttps://graphviz.org/\u003e`_\nand ensure it is in your system's PATH variable:\n\n- For MacOS and Linux users, instructions provided on `GraphViz Downloads \u003chttps://www.graphviz.org/download/\u003e`_ automatically add GraphViz to your path.\n- Windows users may need to manually add GraphViz to your PATH (see `Simplified Windows installation procedure \u003chttps://forum.graphviz.org/t/new-simplified-installation-procedure-on-windows/224\u003e`_).\n- Run ``dot -V`` in Terminal or Command Prompt to verify installation.\n\n\nEnvironment Setup\n-----------------\n\nVirtual environment\n^^^^^^^^^^^^^^^^^^^\n\nWe recommend working in a dedicated environment, e.g., using ``venv``:\n\n.. code-block:: bash\n\n    python -m venv fluxus\n    source fluxus/bin/activate\n\nor ``conda``:\n\n.. code-block:: bash\n\n    conda env create -f environment.yml\n    conda activate fluxus\n\n\nContributing\n------------\n\nContributions to *fluxus* are welcome and appreciated! Please see the\n`Contributing \u003cCONTRIBUTING.md\u003e`_ section for information.\n\n\nLicense\n-------\n\nThis project is under the Apache License 2.0, allowing free use, modification, and distribution with added protections against patent litigation. \nSee the `LICENSE \u003cLICENSE\u003e`_ file for more details or visit `Apache 2.0 \u003chttps://www.apache.org/licenses/LICENSE-2.0\u003e`_.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbcg-x-official%2Ffluxus","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbcg-x-official%2Ffluxus","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbcg-x-official%2Ffluxus/lists"}