{"id":13436524,"url":"https://github.com/sogou/workflow","last_synced_at":"2025-05-14T11:08:16.942Z","repository":{"id":37003784,"uuid":"283410102","full_name":"sogou/workflow","owner":"sogou","description":"C++ Parallel Computing and Asynchronous Networking Framework","archived":false,"fork":false,"pushed_at":"2025-05-06T13:28:31.000Z","size":4803,"stargazers_count":13720,"open_issues_count":19,"forks_count":2490,"subscribers_count":240,"default_branch":"master","last_synced_at":"2025-05-07T10:52:35.285Z","etag":null,"topics":["consul","dag","http","kafka","mysql","redis","sogou","tasking"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/sogou.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-07-29T05:47:01.000Z","updated_at":"2025-05-07T09:00:36.000Z","dependencies_parsed_at":"2023-01-17T12:31:43.519Z","dependency_job_id":"5e87eb93-d7d4-489d-924e-7abb01d36c51","html_url":"https://github.com/sogou/workflow","commit_stats":{"total_commits":1535,"total_committers":46,"mean_commits":"33.369565217391305","dds":0.6918566775244299,"last_synced_commit":"efdba8a0e39700242afd40541af0d0ae2b9ee243"},"previous_names":[],"tags_count":40,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sogou%2Fworkflow","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sogou%2Fworkflow/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sogou%2Fworkflow/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/sogou%2Fworkflow/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/sogou","download_url":"https://codeload.github.com/sogou/workflow/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254129481,"owners_count":22019628,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["consul","dag","http","kafka","mysql","redis","sogou","tasking"],"created_at":"2024-07-31T03:00:49.512Z","updated_at":"2025-05-14T11:08:16.893Z","avatar_url":"https://github.com/sogou.png","language":"C++","readme":"[简体中文版（推荐）](README_cn.md)\n\n## Sogou C++ Workflow\n\n[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://github.com/sogou/workflow/blob/master/LICENSE)\n[![Language](https://img.shields.io/badge/language-c++-red.svg)](https://en.cppreference.com/) \n[![Platform](https://img.shields.io/badge/platform-linux%20%7C%20macos%20%7C%20windows-lightgrey.svg)](https://img.shields.io/badge/platform-linux%20%7C%20macos20%7C%20windows-lightgrey.svg)\n[![Build Status](https://img.shields.io/github/actions/workflow/status/sogou/workflow/ci.yml?branch=master)](https://github.com/sogou/workflow/actions?query=workflow%3A%22ci+build%22++)\n\nAs **Sogou\\`s C++ server engine**, Sogou C++ Workflow supports almost all **back-end C++ online services** of Sogou, including all search services, cloud input method, online advertisements, etc., handling more than **10 billion** requests every day. This is an **enterprise-level programming engine** in light and elegant design which can satisfy most C++ back-end development requirements.\n\n#### You can use it:\n\n* To quickly build an **HTTP server**:\n\n~~~cpp\n#include \u003cstdio.h\u003e\n#include \"workflow/WFHttpServer.h\"\n\nint main()\n{\n    WFHttpServer server([](WFHttpTask *task) {\n        task-\u003eget_resp()-\u003eappend_output_body(\"\u003chtml\u003eHello World!\u003c/html\u003e\");\n    });\n\n    if (server.start(8888) == 0) { // start server on port 8888\n        getchar(); // press \"Enter\" to end.\n        server.stop();\n    }\n\n    return 0;\n}\n~~~\n\n* As a **multifunctional asynchronous client**, it currently supports `HTTP`, `Redis`, `MySQL` and `Kafka` protocols.\n  * ``MySQL`` protocol supports ``MariaDB``, ``TiDB`` as well.\n* To implement **client/server on user-defined protocol** and build your own **RPC system**.\n  * [srpc](https://github.com/sogou/srpc) is based on it and it is an independent open source project, which supports srpc, brpc, trpc and thrift protocols.\n* To build **asynchronous workflow**; support common **series** and **parallel** structures, and also support any **DAG** structures.\n* As a **parallel computing tool**. In addition to **networking tasks**, Sogou C++ Workflow also includes **the scheduling of computing tasks**. All types of tasks can be put into **the same** flow.\n* As an **asynchronous file IO tool** in `Linux` system, with high performance exceeding any system call. Disk file IO is also a task.\n* To realize any **high-performance** and **high-concurrency** back-end service with a very complex relationship between computing and networking.\n* To build a **micro service** system.\n  * This project has built-in **service governance** and **load balancing** features.\n* Wiki link : [PaaS Architecture](https://github.com/sogou/workflow/wiki)\n\n#### Compiling and running environment\n\n* This project supports `Linux`, `macOS`, `Windows`, `Android` and other operating systems.\n  * `Windows` version is currently released as an independent [branch](https://github.com/sogou/workflow/tree/windows), using `iocp` to implement asynchronous networking. All user interfaces are consistent with the `Linux` version.\n* Supports all CPU platforms, including 32 or 64-bit `x86` processors, big-endian or little-endian `arm` processors, `loongson` processors.\n* Master branch requires SSL and `OpenSSL 1.1` or above is recommended. Fully compatible with BoringSSL. If you don't like SSL, you may checkout the [nossl](https://github.com/sogou/workflow/tree/nossl) branch.\n* Uses the `C++11` standard and therefore, it should be compiled with a compiler which supports `C++11`. Does not rely on `boost` or `asio`.\n* No other dependencies. However, if you need `Kafka` protocol, some compression libraries should be installed, including `lz4`, `zstd` and `snappy`.\n\n### Get started (Linux, macOS):\n~~~sh\ngit clone https://github.com/sogou/workflow\ncd workflow\nmake\ncd tutorial\nmake\n~~~~\n\n#### With SRPC Tool (NEW!)：\nhttps://github.com/sogou/srpc/blob/master/tools/README.md\n\n#### With [apt-get](https://launchpad.net/ubuntu/+source/workflow) on Debian Linux, ubuntu:\nSogou C++ Workflow has been packaged for Debian Linux and ubuntu 22.04.  \nTo install the Workflow library for development purposes:\n~~~~sh\nsudo apt-get install libworkflow-dev\n~~~~\n\nTo install the Workflow library for deployment:\n~~~~sh\nsudo apt-get install libworkflow1\n~~~~\n\n#### With [dnf](https://packages.fedoraproject.org/pkgs/workflow) on Fedora Linux:\nSogou C++ Workflow has been packaged for Fedora Linux.  \nTo install the Workflow library for development purposes:\n~~~~sh\nsudo dnf install workflow-devel\n~~~~\n\nTo install the Workflow library for deployment:\n~~~~sh\nsudo dnf install workflow\n~~~~\n\n#### With xmake\n\nIf you want to use xmake to build workflow, you can see [xmake build document](docs/en/xmake.md)\n\n# Tutorials\n\n* Client\n  * [Creating your first task：wget](docs/en/tutorial-01-wget.md)\n  * [Implementing Redis set and get：redis\\_cli](docs/en/tutorial-02-redis_cli.md)\n  * [More features about series：wget\\_to\\_redis](docs/en/tutorial-03-wget_to_redis.md)\n* Server\n  * [First server：http\\_echo\\_server](docs/en/tutorial-04-http_echo_server.md)\n  * [Asynchronous server：http\\_proxy](docs/en/tutorial-05-http_proxy.md)\n* Parallel tasks and Series　\n  * [A simple parallel wget：parallel\\_wget](docs/en/tutorial-06-parallel_wget.md)\n* Important topics\n  * [About error](docs/en/about-error.md)\n  * [About timeout](docs/en/about-timeout.md)\n  * [About global configuration](docs/en/about-config.md)\n  * [About DNS](docs/en/about-dns.md)\n  * [About exit](docs/en/about-exit.md)\n* Computing tasks\n  * [Using the build-in algorithm factory：sort\\_task](docs/en/tutorial-07-sort_task.md)\n  * [User-defined computing task：matrix\\_multiply](docs/en/tutorial-08-matrix_multiply.md)\n  * [Use computing task in a simple way: go task](docs/en/about-go-task.md)\n* Asynchronous File IO tasks\n  * [Http server with file IO：http\\_file\\_server](docs/en/tutorial-09-http_file_server.md)\n* User-defined protocol\n  * [A simple user-defined protocol: client/server](docs/en/tutorial-10-user_defined_protocol.md)\n  * [Use TLV message](docs/en/about-tlv-message.md)\n* Other important tasks/components\n  * [About timer](docs/en/about-timer.md)\n  * [About counter](docs/en/about-counter.md)\n  * [About resource pool](docs/en/about-resource-pool.md)\n  * [About module](docs/en/about-module.md)\n  * [About DAG](docs/en/tutorial-11-graph_task.md)\n* Service governance\n  * [About service governance](docs/en/about-service-governance.md)\n  * [More documents about upstream](docs/en/about-upstream.md)\n* Connection context\n  * [About connection context](docs/en/about-connection-context.md)\n* Built-in clients\n  * [Asynchronous MySQL client：mysql\\_cli](docs/en/tutorial-12-mysql_cli.md)\n  * [Asynchronous Kafka client: kafka\\_cli](docs/en/tutorial-13-kafka_cli.md)\n\n#### Programming paradigm\n\nWe believe that a typical back-end program=protocol+algorithm+workflow and should be developed completely independently.\n\n* Protocol\n  * In most cases, users use built-in common network protocols, such as HTTP, Redis or various rpc.\n  * Users can also easily customize user-defined network protocol. In the customization, they only need to provide serialization and deserialization functions to define their own client/server.\n* Algorithm\n  * In our design, the algorithm is a concept symmetrical to the protocol.\n    * If protocol call is rpc, then algorithm call is an apc (Async Procedure Call).\n  * We have provided some general algorithms, such as sort, merge, psort, reduce, which can be used directly.\n  * Compared with a user-defined protocol, a user-defined algorithm is much more common. Any complicated computation with clear boundaries should be packaged into an algorithm.\n* Workflow\n  * Workflow is the actual business logic, which is to put the protocols and algorithms into the flow graph for use.\n  * The typical workflow is a closed series-parallel graph. Complex business logic may be a non-closed DAG.\n  * The workflow graph can be constructed directly or dynamically generated based on the results of each step. All tasks are executed asynchronously.\n\nBasic task, task factory and complex task\n\n* Our system contains six basic tasks: networking, file IO, CPU, GPU, timer, and counter.\n* All tasks are generated by the task factory and automatically recycled after callback.\n  * Server task is one kind of special networking task, generated by the framework which calls the task factory, and handed over to the user through the process function.\n* In most cases, the task generated by the user through the task factory is a complex task, which is transparent to the user.\n  * For example, an HTTP request may include many asynchronous processes (DNS, redirection), but for user, it is just a networking task.\n  * File sorting seems to be an algorithm, but it actually includes many complex interaction processes between file IO and CPU computation.\n  * If you think of business logic as building circuits with well-designed electronic components, then each electronic component may be a complex circuit.\n\nAsynchrony and encapsulation based on `C++11 std::function`\n\n* Not based on user mode coroutines. Users need to know that they are writing asynchronous programs.\n* All calls are executed asynchronously, and there is almost no operation that occupies a thread.\n  * Although we also provide some facilities with semi-synchronous interfaces, they are not core features.\n* We try to avoid user's derivations, and encapsulate user behavior with `std::function` instead, including:\n  * The callback of any task.\n  * Any server's process. This conforms to the `FaaS` (Function as a Service) idea.\n  * The realization of an algorithm is simply a `std::function`. But the algorithm can also be implemented by derivation.\n\nMemory reclamation mechanism\n\n* Every task will be automatically reclaimed after the callback. If a task is created but a user does not want to run it, the user needs to release it through the dismiss method.\n* Any data in the task, such as the response of the network request, will also be recycled with the task. At this time, the user can use `std::move()` to move the required data.\n* SeriesWork and ParallelWork are two kinds of framework objects, which are also recycled after their callback.\n  * When a series is a branch of a parallel, it will be recycled after the callback of the parallel that it belongs to.\n* This project doesn’t use `std::shared_ptr` to manage memory.\n\n#### Any other questions?\n\nYou may check the [FAQ](https://github.com/sogou/workflow/issues/406) and [issues](https://github.com/sogou/workflow/issues) list first to see if you can find the answer.\n\nYou are very welcome to send the problems you encounter in use to [issues](https://github.com/sogou/workflow/issues), and we will answer them as soon as possible. At the same time, more issues will also help new users.\n\n","funding_links":[],"categories":["Frameworks","C++","NetWork","网络服务","Service Toolkits","Networking","Library (embedded usage)","Software"],"sub_categories":["网络服务_其他","C++"],"project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsogou%2Fworkflow","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fsogou%2Fworkflow","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fsogou%2Fworkflow/lists"}