{"id":13655968,"url":"https://github.com/kkyon/botflow","last_synced_at":"2025-04-23T17:30:54.963Z","repository":{"id":55425664,"uuid":"145360265","full_name":"kkyon/botflow","owner":"kkyon","description":"Python Fast Dataflow programming  framework for Data pipeline work( Web Crawler,Machine Learning,Quantitative Trading.etc)","archived":false,"fork":false,"pushed_at":"2020-12-31T09:03:22.000Z","size":2193,"stargazers_count":1201,"open_issues_count":4,"forks_count":102,"subscribers_count":51,"default_branch":"master","last_synced_at":"2025-04-14T07:18:09.921Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"http://botflow.readthedocs.io","language":"Python","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"other","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/kkyon.png","metadata":{"files":{"readme":"README.rst","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2018-08-20T03:13:31.000Z","updated_at":"2025-03-13T18:49:44.000Z","dependencies_parsed_at":"2022-08-15T00:01:02.857Z","dependency_job_id":null,"html_url":"https://github.com/kkyon/botflow","commit_stats":null,"previous_names":["kkyx/databot"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kkyon%2Fbotflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kkyon%2Fbotflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kkyon%2Fbotflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kkyon%2Fbotflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/kkyon","download_url":"https://codeload.github.com/kkyon/botflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":250480337,"owners_count":21437524,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-02T04:00:44.060Z","updated_at":"2025-04-23T17:30:53.913Z","avatar_url":"https://github.com/kkyon.png","language":"Python","readme":"=======\nBotflow\n=======\n0.2.0 alpha\n\n\n* Dataflow programming framework\n* Paralleled in coroutines and ThreadPool\n* Type- and content-based route function\n* Interactive programming with Jupyter Notebook\n\nRequirements\n------------\n`Graphviz \u003chttp://brewformulas.org/Graphviz\u003e`_.\n\nInstalling\n----------\n\nInstall and update using ``pip``:\n\n`pip install -U botflow`\n\nDocumentation\n------------\n\nhttp://botflow.readthedocs.io\n\n\n\n\nWhat's dataflow programming?\n===============================\n\nAll functions are connected by pipes (queues) and communicate by data.  \n\nWhen data come in, the function will be called and return the result.\n\nThink about the pipeline operation in unix: ``ls|grep|sed``.\n\nBenefits:\n\n#. Decouple data and functionality\n#. Easy to reuse \n\nBotflow provides pipe and route. It makes dataflow programming and powerful data flow processes easier.\n\n\nBotflow is...\n=============\n\n- **Simple**\n\nBotflow is easy to use and maintain, *does not need configuration files*, and knows about ``asyncio`` and how to parallelize computation.\n\nHere's one of the simple applications you can make:\n\n_Load the price of Bitcoin every 2 seconds. Advantage price aggregator sample can be found `here \u003chttps://github.com/kkyon/Botflow/tree/master/examples\u003e`_.\n\n\n.. code-block:: python\n\n    from botflow import *\n\n\n    def main():\n        Pipe(\n\n            Timer(delay=2),  # send timer data to pipe every 2 seconds\n            \"http://api.coindesk.com/v1/bpi/currentprice.json\",  # send url to pipe when timer trigger\n            HttpLoader(),  # read url and load http response\n            lambda r: r.json['bpi']['USD']['rate_float'],  # read http response and parse as json\n            print,  # print out\n\n        )\n\n        Bot.render('ex_output/simple_bitcoin_price')\n        Bot.run()\n    main()\n\n**Or write in chain style**\n\n\n.. code-block:: python\n\n\n\n    from botflow import *\n    p_cd_bitcoin=Pipe().Timer(delay=2).Loop(\"http://api.coindesk.com/v1/bpi/currentprice.json\")\\\n                .HttpLoader().Map(lambda r: r.json['bpi']['USD']['rate_float']).Map(print)\n\n    p_cd_bitcoin.run()\n\n\n\n \n- **Flow Graph**\nWith render function:\n`Bot.render('bitcoin_arbitrage')`\nBotflow will render the data flow network into a graphviz image.\nbelow is the flow graph generated by Botflow.Aggreate 6 exchanges bitcoin price for trading.\n\n\n.. image:: docs/bitcoin_arbitrage.png\n    :width: 400  \n\n\n \n\n- **Fast**\nNodes will be run in parallel, and they will perform well when processing stream data.\n:Web Crawle: Botflow is 10x fatter than Scrapy\n\n\n \n\n- **Replay-able**\n\nWith replay mode enabled:\n``config.replay_mode=True``\nwhen an exception is raised at step N, you don't need to run from setup 1 to N.\nBotflow will replay the data from nearest completed node, usually step N-1.\nIt will save a lot of time in the development phase.\n\nRelease\n\n:**0.2.0**: Milestone release.:\n\n            # Jupyter support. Able to run inside Jupyter note book.\n\n            # pipe can be nest in another Pipe.\n\n\n            p1=Pipe(get_image)\n            p2=Pipe(get_price)\n            p_get_all=Pipe(Zip(p1,p2)).Filter\n\n            # Support Chain style pipe line creating.\n\n                Pipe(range(1,10)).Map(lambda x:x+1).Fiter(lambda x:x\u003e2)\n\n                same as :\n\n                Pipe(range(1,10),lambda x:x+1,Filter(lambda x:x\u003e2))\n\n\n\n:**0.1.9**: Major change see below .:\n\n            # Backpressure rate limit support\n\n            # Httpserver support\n\n            # new Node support. *Zip*, *SendTo* *Flat* for make loop and redirect the flow\n\n            # Type hints support .for function type route\n\n            # reorge the source code for readable.\n\n\n:**0.1.8**: http://docs.botflow.org/en/latest/change/0.1.8.html .:\n            \n            #. Support parallel in ThreadPool for slow function.\n            \n            #. Loop Node  is  deprecated. raw value and Iterable value can be used directly.\n            \n            #. improve performance of BlockedJoin\n            \n:**0.1.7**: \n\n\nRoadMap\n=======\n- Will add Httpserver support(REST,Websocket).  \n- Will support server machine learning Model online.\n- Finshe the api reference doc.\n- Rename project to Botflow.?\n\nMore about Botflow\n===============\n\nData-\nprogramming is typically applied to streams of structured data for filtering, transforming, aggregating (such as computing statistics), or calling other programs.\n\nBotflow has a few basic concepts to implement Dataflow programming .\n\n- **Source**\n        It is feed stream data to the pipe.\n\n    * **Timer**: It will send a message in the pipe by timer param. **delay**, **max_time** **until** some finished\n    * **Pipe.run**: you can use Pipe.run to trigger the data into pipe. By default it will feed int **0**\n\n\n\n- **Function**\n        It is callable unit.Any callable function and object can work as Node. It is driven by data. Custom functions work as Map unit.\n        There are some built-in nodes:\n\n   \n\n   * **Fetch**: (Alias:HttpLoader)  Get a url and return the HTTP response\n   * **AioFile**: for file I/O.\n   * **SpeedLimit**: limit the stream speed limit\n   * **Delay**: delay in special second.\n   * **Map**  : Work ad Convert unit.\n   * **Filter** : Drop data from pipe if it does not match some condition\n   * **Flat** : Drop data from pipe if it does not match some condition\n\n\n- **Route**\n        It will be used to create a complex data flow network, not just one main process. Botflow can nest Routes inside Routes.\n        It is a powerful concept.\n        There are some pre built-in Route:\n    * **Pipe**: It is the main stream process of the program. All units will work inside.\n    * **Tee** : (Alias:Branch) Duplicate data from parent pipe to a child pipe as branch.\n    * **Zip** :   Combine multi pipes result to list.\n    * **Link**: (Alias: LinkTo)  Route flow to any Node or Route for making loop , circle\n\n\nAll units (Pipe, Node, Route) communicate via queues and perform parallel computation in coroutines.\nThis is abstracted so that Botflow can be used with only limited knowledge of ``asyncio``.\n\n\n      \n\nContributing\n------------\n\n\nDonate\n------\n\n\nLinks\n-----\n","funding_links":[],"categories":["数据管道和流处理","Python","Data Pipelines \u0026 Streaming","Uncategorized"],"sub_categories":["Uncategorized"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkkyon%2Fbotflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fkkyon%2Fbotflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fkkyon%2Fbotflow/lists"}