{"id":13831619,"url":"https://github.com/cloudwego/kitex-benchmark","last_synced_at":"2025-04-06T05:16:22.709Z","repository":{"id":38831721,"uuid":"392626222","full_name":"cloudwego/kitex-benchmark","owner":"cloudwego","description":null,"archived":false,"fork":false,"pushed_at":"2025-03-27T03:55:53.000Z","size":2791,"stargazers_count":54,"open_issues_count":4,"forks_count":29,"subscribers_count":14,"default_branch":"main","last_synced_at":"2025-03-30T02:06:06.888Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Go","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cloudwego.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":null,"code_of_conduct":"CODE_OF_CONDUCT.md","threat_model":null,"audit":null,"citation":null,"codeowners":".github/CODEOWNERS","security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-08-04T09:22:50.000Z","updated_at":"2025-03-24T05:47:20.000Z","dependencies_parsed_at":"2024-03-14T07:44:20.717Z","dependency_job_id":"ab4f557a-c41a-4775-9f0e-b140fb25e460","html_url":"https://github.com/cloudwego/kitex-benchmark","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudwego%2Fkitex-benchmark","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudwego%2Fkitex-benchmark/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudwego%2Fkitex-benchmark/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cloudwego%2Fkitex-benchmark/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cloudwego","download_url":"https://codeload.github.com/cloudwego/kitex-benchmark/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":247436286,"owners_count":20938533,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-08-04T10:01:33.612Z","updated_at":"2025-04-06T05:16:22.684Z","avatar_url":"https://github.com/cloudwego.png","language":"Go","funding_links":[],"categories":["Go"],"sub_categories":[],"readme":"# kitex-benchmark\n\nEnglish | [中文](README_cn.md)\n\nThis project shows several simple uses of [kitex][kitex] and provides several comparison projects.\n\nDue to the differences in the protocols and transmission modes used by different frameworks, it's difficult to benchmark them all under the same baseline. [kitex][kitex] gives several simple combinations for reference.\n\n1. [kitex][kitex]:\n\t- Multi-Message Protocol: Thrift(recommended), KitexProtobuf(Customized Protobuf Message Protocol), GRPC(Same with gRPC)\n\t- Multi-Transmission Mode: long connection pool (recommended), connection multiplexing (mux)\n2. Comparison Frameworks:\n\t- [Thrift][thrift]: Kitex is the only full-featured Thrift Golang Framework at now.\n\t- [Protobuf][protobuf]: [grpc][grpc], [rpcx][rpcx], [arpc][arpc] (all use connection multiplexing), but note that there are differences in message protocols.\n\n## Usage\n\n### Localhost Mode\n\nPlease make sure to meet [Requirements](#Requirements) before execution.\n\n#### Thrift\n\n```bash\n./scripts/benchmark_thrift.sh\n```\n\n#### Protobuf\n\n```bash\n./scripts/benchmark_pb.sh\n```\n\n#### GRPC\n\n```bash\n./scripts/benchmark_grpc.sh\n```\n\n### External Network Mode\n\nThe packets in loopback network mode don't enter the network card, failing to truly simulate the online services communication. So it also provides an approach to bench the\nclient and server individually.\n\nBut it should be noted that if the host machine has more than the CPU cores set by taskset, the process will borrow other ksoftirqd kernel threads that are not controlled by taskset, and shares the computation of other CPUs. Therefore, it is recommended to use the same machine specification as taskset, or delete taskset when you use cross-node mode.\n\n#### Thrift\n\n```bash\n# host A\n./scripts/run_thrift_servers.sh\n\n# host B\n./scripts/run_thrift_clients.sh\n```\n\n#### Protobuf\n\n```bash\n# host A\n./scripts/run_pb_servers.sh\n\n# host B\n./scripts/run_pb_clients.sh\n```\n\n#### GRPC\n\n```bash\n# host A\n./scripts/run_grpc_servers.sh\n# host B\n./scripts/run_grpc_clients.sh\n```\n\n### Benchmark Diff\n\nAll benchmark result will be written at `./output`, and named as current time by default(or $REPORT):\n\n```bash\n$ scripts/benchmark_thrift.sh\n$ ls output/\n2021-12-13-21-40.log # raw output log\n2021-12-13-21-40.csv # processed data\n\n$ REPORT=feat-gopool scripts/benchmark_thrift.sh\n$ ls output/\nfeat-gopool.log # raw output log\nfeat-gopool.csv # processed data\n```\n\nDiff two different benchmark results' csv files:\n\n```bash\n# Usage: python3 ./scripts/reports/diff.py target.csv current.csv\n\npython3 ./scripts/reports/diff.py output/2021-12-13-21-40.csv output/2021-12-13-21-44.csv\n\n# output:\n# [KITEX-MUX]   100            1024           275604.66(+0.4%)     1.13(+0.0%)     2.01(-0.5%)\n# [KITEX]       100            1024           218999.03(-0.4%)     1.28(-3.0%)     3.73(-2.1%)\n```\n\n### Profiling\n\nSince the default benchmark will complete quickly, to obtain enough time to do profiling, you can increase the parameter `n` in `./scripts/base.sh`.\n\n#### Profiling Client\n\n```bash\ngo tool pprof localhost:18888/debug/pprof/{pprof_type}\n```\n\n#### Profiling Server\n\nFind port mapping of different servers at the corresponding script, such as:\n\n```bash\ncat ./scripts/benchmark_pb.sh\n\n# ...\nrepo=(\"grpc\" \"kitex\" \"kitex-mux\" \"rpcx\" \"arpc\" \"arpc-nbio\")\nports=(8000 8001 8002 8003 8004 8005)\n```\n\nAfter obtaining the corresponding server port number, execute:\n\n```bash\ngo tool pprof localhost:{port}/debug/pprof/{pprof_type}\n```\n\n### More scenarios\n\nModify the `./scripts/base.sh` file:\n\n```bash\n# Send pressure test request number\nn=5000000\n# Request body size\nbody=(1024 5120)\n# Concurrency\nconcurrent=(100 200 400 600 800 1000)\n# server handler sleep time (/ms), the default is 0\nsleep=0\n```\n\n## Requirements\n\n- OS: Linux\n  * By default, it depends on the command `taskset` to limit the CPUs used by the client and server; if it is executed on other systems, please modify the script.\n- CPU: Recommended \u003e=20 cores, minimum \u003e=4 cores\n  * The benchmark script requires 20 CPUs by default, which can be modified or deleted in the `taskset -c ...` part of the script.\n\n## Reference Data\n\n**Notes:**\n\nThe benchmark ensures the caller has sufficient machine resources **overwhelming the server**, and focuses more on server performance. The performance data of the caller will be provided later.\n\n### Specification\n\n* CPU: Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz\n    * server 4-CPUs, client 16-CPUs\n* OS: Debian 5.4.56.bsk.1-amd64 x86_64 GNU/Linux\n* Go: 1.17.2\n\n### Data (echo size 1KB)\n\n- [Thrift Raw Data](scripts/reports/thrift.csv)\n- [Protobuf Raw Data](scripts/reports/pb.csv)\n- [GRPC Raw Data](scripts/reports/grpc.csv)\n\n#### Thrift\n\n![image](docs/images/thrift_qps.png)\n![image](docs/images/thrift_tp99.png)\n![image](docs/images/thrift_tp999.png)\n\n#### Protobuf\nNote: \n1. The message protocols used by each framework are different. About GRPC, the next part has comparison.\n2. Some frameworks (such as ARPC) use multiplexing, so should refer to the result of KITEX-MUX for comparison.\n\n![image](docs/images/pb_qps.png)\n![image](docs/images/pb_tp99.png)\n![image](docs/images/pb_tp999.png)\n\n#### GRPC\n\n![image](docs/images/grpc_qps.png)\n![image](docs/images/grpc_tp99.png)\n![image](docs/images/grpc_tp999.png)\n\n#### Streaming\n\n![image](docs/images/streaming_qps.png)\n![image](docs/images/streaming_tp99.png)\n![image](docs/images/streaming_tp999.png)\n\n[kitex]: https://github.com/cloudwego/kitex\n[grpc]: https://github.com/grpc/grpc\n[rpcx]: https://github.com/smallnest/rpcx\n[arpc]: https://github.com/lesismal/arpc\n[thrift]: https://thrift.apache.org\n[protobuf]: https://developers.google.com/protocol-buffers/docs/gotutorial\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudwego%2Fkitex-benchmark","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcloudwego%2Fkitex-benchmark","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcloudwego%2Fkitex-benchmark/lists"}