{"id":13737754,"url":"https://github.com/maki-nage/makinage","last_synced_at":"2025-12-14T22:50:24.627Z","repository":{"id":41886908,"uuid":"244487574","full_name":"maki-nage/makinage","owner":"maki-nage","description":"Stream Processing Made Easy","archived":false,"fork":false,"pushed_at":"2022-04-24T20:17:10.000Z","size":232,"stargazers_count":39,"open_issues_count":8,"forks_count":1,"subscribers_count":4,"default_branch":"master","last_synced_at":"2024-11-12T07:56:01.746Z","etag":null,"topics":["distributed-systems","kafka","machine-learning","python","reactive-machine-learning","reactive-programming","reactive-systems","stream-processing","streaming","streaming-data"],"latest_commit_sha":null,"homepage":"https://www.makinage.org","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/maki-nage.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2020-03-02T22:18:22.000Z","updated_at":"2024-10-22T17:06:40.000Z","dependencies_parsed_at":"2022-08-11T20:10:57.747Z","dependency_job_id":null,"html_url":"https://github.com/maki-nage/makinage","commit_stats":null,"previous_names":[],"tags_count":11,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maki-nage%2Fmakinage","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maki-nage%2Fmakinage/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maki-nage%2Fmakinage/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/maki-nage%2Fmakinage/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/maki-nage","download_url":"https://codeload.github.com/maki-nage/makinage/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":224742329,"owners_count":17362229,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["distributed-systems","kafka","machine-learning","python","reactive-machine-learning","reactive-programming","reactive-systems","stream-processing","streaming","streaming-data"],"created_at":"2024-08-03T03:01:59.751Z","updated_at":"2025-12-14T22:50:24.575Z","avatar_url":"https://github.com/maki-nage.png","language":"Python","funding_links":[],"categories":["Python","Table of Contents"],"sub_categories":["Streaming Engine"],"readme":"===========================\n|makinage-logo| Maki Nage\n===========================\n\n.. |makinage-logo| image:: https://github.com/maki-nage/makinage/raw/master/asset/makinage_logo.png\n\nStream Processing Made Easy\n\n.. image:: https://badge.fury.io/py/makinage.svg\n    :target: https://badge.fury.io/py/makinage\n\n.. image:: https://github.com/maki-nage/makinage/workflows/Python%20package/badge.svg\n    :target: https://github.com/maki-nage/makinage/actions?query=workflow%3A%22Python+package%22\n    :alt: Github WorkFlows\n\n.. image:: https://github.com/maki-nage/makinage/raw/master/asset/docs_download.svg\n    :target: https://www.makinage.org/doc/makinage-book/latest/index.html\n    :alt: Documentation\n\n\nMaki Nage is a Python stream processing library and framework. It provides\n**expressive** and **extensible** APIs. Maki Nage speeds up the development of\nstream applications. It can be used to process **stream** and **batch** data.\nMore than that, it allows to develop an application with batch data, and deploy\nit as a **Kafka micro-service**.\n\n`Read the doc \u003chttps://www.makinage.org/doc/makinage-book/latest/index.html\u003e`_\nto learn more.\n\n.. image:: https://github.com/maki-nage/makinage/raw/master/asset/graph.png\n    :width: 50%\n\nMain Features\n==============\n\n* **Expressive** and **Extensible** APIs: Maki-Nage is based on `ReactiveX \u003chttps://github.com/ReactiveX/RxPY\u003e`_.\n* Deployment Ready: Maki-Nage uses **Kafka** to scale the workload, and be resilient to errors.\n* **Unifies** Batch and Stream processing: The same APIs work on both sources of data.\n* Flexible: Start working on your laptop, continue on a server, deploy on a cluster.\n* **ML Streaming Serving**: Serve your machine learning model as a Kafka micro-service.\n\nInstallation\n==============\n\nMaki Nage is available on PyPI:\n\n.. code:: console\n\n    pip install makinage\n\n\nGetting started\n===============\n\nWrite your data transforms\n---------------------------\n\n.. code:: Python\n\n    import rx\n    import rxsci as rs\n\n    def rolling_mean():\n        return rx.pipe(            \n            rs.data.roll(window=3, stride=3, pipeline=rx.pipe(\n                rs.math.mean(reduce=True),\n            )),\n        )\n\nTest your code on batch data\n-------------------------------\n\nYou can test your code from any python data or CSV file.\n\n.. code:: Python\n\n    data = [1, 2, 3, 4, 5, 6, 7]\n\n    rx.from_(data).pipe(\n        rs.state.with_memory_store(rx.pipe(\n            rolling_mean(),\n        )),\n    ).subscribe(\n        on_next=print\n    )\n\n.. code:: console\n\n    2.0\n    5.0\n\n\nDeploy your code as a Kafka micro-service\n-------------------------------------------\n\nTo deploy the code, package it as a function:\n\n.. code:: Python\n\n    def my_app(config, data):\n        roll_mean = rx.from_(data).pipe(\n            rs.state.with_memory_store(rx.pipe(\n                rolling_mean(),\n            )),\n        )\n\n        return roll_mean,\n\nCreate a configuration file:\n\n.. code:: yaml\n\n    application:\n        name: my_app\n    kafka:\n        endpoint: \"localhost\"\n    topics:\n        - name: data\n        - name: features\n    operators:\n        compute_features:\n            factory: my_app:my_app\n            sources:\n                - data\n            sinks:\n                - features\n\nAnd start it!\n\n.. code:: console\n\n    makinage --config myconfig.yaml\n\n\nServe Machine Learning Models\n===============================\n\nMaki Nage contains a model serving tool. With it, serving a machine\nlearning model in streaming mode just requires a configuration file:\n\n.. code:: yaml\n\n    application:\n        name: my_model_serving\n    Kafka:\n        endpoint: \"localhost\"\n    topics:\n    - name: data\n      encoder: makinage.encoding.json\n    - name: model\n      encoder: makinage.encoding.none\n      start_from: last\n    - name: predict\n      encoder: makinage.encoding.json\n    operators:\n      serve:\n        factory: makinage.serve:serve\n        sources:\n          - model\n          - data\n        sinks:\n          - predict\n    config:\n      serve: {}\n\nAnd then serving the model it done the same way than any makinage application:\n\n.. code:: console\n\n    makinage --config config.serve.yaml\n\n\nSome pre and post processing steps are possible if input features or predictions\nmust be modified before/after the inference:\n\n.. image:: https://github.com/maki-nage/makinage/raw/master/asset/serve.png\n\n`Read the book \u003chttps://www.makinage.org/doc/makinage-book/latest/serving.html#\u003e`_\nto learn more.\n\n\nPublications\n===============\n\n* Toward Data Science: `Stream Processing Made Easy \u003chttps://towardsdatascience.com/stream-processing-made-easy-5f4892736623\u003e`_\n* KDnuggets: `Real-Time Histogram Plots on Unbounded Data \u003chttps://www.kdnuggets.com/2021/09/real-time-histogram-plots-unbounded-data.html\u003e`_\n\nLicense\n=========\n\nMaki Nage is publised under the MIT License.","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaki-nage%2Fmakinage","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmaki-nage%2Fmakinage","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmaki-nage%2Fmakinage/lists"}