{"id":18096080,"url":"https://github.com/newplan/rdma_comm_core","last_synced_at":"2025-04-13T10:05:03.257Z","repository":{"id":45625632,"uuid":"407440472","full_name":"NEWPLAN/rdma_comm_core","owner":"NEWPLAN","description":"C++-based rdma communication library for easy use","archived":false,"fork":false,"pushed_at":"2024-05-29T02:56:43.000Z","size":320,"stargazers_count":7,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2025-03-27T01:23:51.446Z","etag":null,"topics":["rdma"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/NEWPLAN.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-09-17T07:04:03.000Z","updated_at":"2025-01-15T12:49:14.000Z","dependencies_parsed_at":"2024-10-31T19:12:35.189Z","dependency_job_id":"6fae9279-c9ec-4be3-bac2-514b7bf0c0d1","html_url":"https://github.com/NEWPLAN/rdma_comm_core","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NEWPLAN%2Frdma_comm_core","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NEWPLAN%2Frdma_comm_core/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NEWPLAN%2Frdma_comm_core/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/NEWPLAN%2Frdma_comm_core/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/NEWPLAN","download_url":"https://codeload.github.com/NEWPLAN/rdma_comm_core/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":248695438,"owners_count":21146954,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["rdma"],"created_at":"2024-10-31T19:12:29.792Z","updated_at":"2025-04-13T10:05:03.233Z","avatar_url":"https://github.com/NEWPLAN.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Introduction\n\n[rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) is a wrapper of rdma communication primitives implemented in the C++ programming language, which includes `SEND`, `RECV`, `WRITE`, `READ`, `WRITE_WITH_IMM`. It support multiple `work request` processed concurrently to maximun the performance.\n\n[rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) works for the Linux platform and tested on Ubuntu 1604.\n\n\n# Basic Design\n\n## Event Mode\nIn general, RDMA works in the asynchronous event-driven fashion for high performance, and [rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) supports three basic working modes. \n\n- **`polling-aggressively`**: In this mode, thread(s) would proactively poll the `completion queue` without blocking to fetch the executing result (success or failure) of work requests submitted before. Application working in this mode may be latency-sensitive, accordingly, CPU resources are required.\n- **`completion channel`**: In this mode, the thread(s) to fetch the results would be stuck in the function ([ibv_get_cq_event](https://github.com/linux-rdma/rdma-core/blob/aa0fda8580149bdaf5138d7a2891011c10fe6701/libibverbs/verbs.h#L2829)) until waked up by a new incoming event(success or failure). Application working in this mode can save many CPU resources, accordingly, the processing latency increases.\n- **`epoll`**: In this mode, users can set a global `epoll` resource to listen to the event for a group of `completion queues` while not being blocked by the API. By setting a timeout, the polling thread can be released to process other workflows despite no new incoming events.\n\n## Thread Mode \n[rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) is based on thread mode. Currently, it supports two thread modes as follows:\n- **`Per-thread-per-QPair`**: In this mode, assign a private thread to each QPair to track the work request state in their completion queues.\n- **`Shared-CQ`**: In this mode, multiple QPairs may share a global `completion queue(CQ)` to track the ```wqe``` executing status.\n\nThe Thread Mode is configured in [void lazy_config_hca()](https://github.com/NEWPLAN/rdma_comm_core/blob/abf254743b137640849ff5fc8dffa524102e956f/include/rdma_session.h#L51), which should be implemented by sub-class.\n\n\n## Architecture Overview\nThe architecture overview of [rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) is shown below.\n\n![Architecture Overview](docs/arch_overview.jpg)\n\n There are two roles distinguished in [rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core).\n\n- ```ServerSession``` is inherited from ```RDMASession``` and handles the workflow of server sides, such as listen to and accept new connection requests. Once a new connection request is detected, the ```ServerSession``` would allocate corresponding resources and interacts with the client side to establish the connection.\n- ```ClientSession``` is inherited from ```RDMASession``` and handles the workflow of client sides. It interacts with ```ServerSession```  to finish the connection.\n\nBoth ```ServerSession```  and ```ClientSession``` use ```EndPoint``` to identify the connection. Differently, ```ServerSession```  may have multiple ```EndPoint```s to deal with different connections individually, while there is only an ```EndPoint``` in the ```ClientSession``` to deal with data exchange with the server side.\n\n## Dive into an ```EndPoint```\n\n[rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) allocates an individual ```EndPoint``` to manage each connection in both ```ClientSession``` and ```ServerSession``` once a connection request is generated/detected.\nBelow is the inside of an ```EndPoint```.\n![Inside of an EndPoint](docs/inside-endpoint.jpg)\n\n## Basic Components\nBefore showing how EndPoint works, we would introduce each component first.\n- ```RDMADevice``` is the abstraction of RDMA device, and each instance is bound to the hardware device(e.g., NIC) and provides basic services, such as ```query/modify device attributes/status```, ```SEND```, ```RECV```, ```WRITE(-WITH_IMM)```, ```READ```, manage the context/resources of hardware, etc.\n- ```RDMAAdapter``` is the lower abstraction of an RDMA connection context. It is instanced once a connection is launched and interacts with ```RDMADevice``` to process and connection workflow.\n- ```RDMAChannel``` is the higher abstraction of an RDMA connection context. It is inherited from ```RDMAAdapter``` and interacts with applications to process and user-space workflow.\n- ```TCPConnector``` is a wrapper of ```TCP Socket``` and serves as a helper to exchange necessary information when establishing the RDMA connection.\n- ```RDMABuffer``` is the memory manager exposed to the upper applications to provide placeholders for communication workflow.\n\n## Organization\n- There is at least one ```TCPConnector``` owned by an ```EndPoint``` as a helper when establishing the connections. \n- Additionally, there may be one or more ```RDMAChannel``` in an ```EndPoint``` to address different communication services.\n- All communication requests from the upper application are submitted to ```RDMAAdapter``` by ```RDMAChannel``` and finally processed by ```RDMADevice```.\n\n\n# Installation and Usages\nThere are three examples included in the repository, located in the subdirectory named `example`.\n## Requirements\n[rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) uses customized log and backtraces that may require installation before use. All related third libraries are listed as follows.\n- ```cmake``` is used as the project management system, [rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) requires a [cmake](https://github.com/Kitware/CMake/releases/download/v3.21.3/cmake-3.21.3.tar.gz) with version(\u003e=3.18) to simplify the compiling process.\n- ```google-glog``` is a popular log system. [rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) uses the ```RLOG``` customized from [google-glog](https://github.com/google/glog) to show the execution process with different levels (specified by ```RCL_MAX_VLOG_LEVEL=XXX```)\n- ```backward-cpp``` is a popular backtracing subsystem. [rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) uses ```backtrace_service``` customized from [backward-cpp](https://github.com/bombela/backward-cpp) to address the runtime exception.\n- ```gcc/g++``` is used in [rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core) as the default compiler. Particularly, the version of ```gcc/g++``` should be \u003e= [9.3.0](https://ftp.gnu.org/gnu/gcc/gcc-9.3.0/) to support the ```C++``` grammar used in [rdma_comm_core](https://github.com/NEWPLAN/rdma_comm_core).\n  \n## Examples\n- ```ping_pong_test``` is a connection test for any rdma connection pair. It uses default processing workflows on server and client sides.\n- ```lat_bw_benchmark``` is a customized benchmark to evaluate the latency and throughput of RDMA communication primitives (including ```SEND```, ```RECV```, ```WRITE```, ```READ```), by re-implementing the ```RDMAClientSession``` and ```RDMAServerSession```.\n- ```mesh_comm_service``` is a full-mesh communication test among N nodes, where each has ```N-1``` ```ClientSession``` and a ```ServerSession``` to WRITE/RECV data simultaneously.\n\n## Run Example \nWe Run the ```lat_bw_benchmark``` as the example to show the whole workflow:\n\n### compile\n```shell\nmkdir build; cd build; cmake .. -GNinja; ninja;\n```\n### Run Server\n```shell\n~/t/F/3/r/build (main)\u003e RCL_MAX_VLOG_LEVEL=3 ./lat_bw_benchmark --role=master\nRCL_MAX_VLOG_LEVEL is set to 3, indicating level in VLOG (level) less than 3 would show\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nI0923 18:30:27.148751 17905 rdma_session.cc:36] Creating a new RDMASession for\nI0923 18:30:27.149014 17905 rdma_server_sess.cc:15] Creating a new RDMAServerSession for ConnectionTest\nI0923 18:30:27.149031 17905 lat_bw_benchmark.cc:523] Creating SchedulingServer\nI0923 18:30:27.149052 17905 tcp_connector.cc:118] Preparing socket for RDMAServerSession\nI0923 18:30:27.149076 17905 tcp_connector.cc:123] Creating a socket: 3 for RDMAServerSession\n[INFO] Having modified IP ToS for socket: 3 from 0x0(old) to 0x10(new)\n[INFO] Having modified IP priority for socket: 3 from 6(old) to 4(new)\n[INFO] Having set IP port resue\nI0923 18:30:27.149133 17905 tcp_connector.cc:128] [Done] Preparing socket...\nI0923 18:30:27.149145 17905 rdma_server_sess.cc:29] RDMAServer is preparing everything for accepting new connections\nI0923 18:30:27.149181 17905 rdma_server_sess.cc:46] The server is listening on 2020\nI0923 18:30:32.648588 17905 rdma_session.cc:156] Handles new connect request by @ConnectionTest\nI0923 18:30:32.648682 17905 rdma_session.cc:159] the remote info of connection request is 12.12.12.113:60878\nI0923 18:30:32.648783 17905 rdma_endpoint.cc:23] Create RDMAEndPoint\nI0923 18:30:32.648993 17905 rdma_endpoint.cc:29] Connection Test OK for TCPConnector@RDMAEndPoint@[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nI0923 18:30:32.649061 17905 rdma_endpoint.cc:91] creating\u0026preparing an rdma channel for DataChannel\nI0923 18:30:32.649116 17905 rdma_adapter.cc:20] Creating a (virtual) adapter for DataChannel\nI0923 18:30:32.649149 17905 rdma_channel.cc:10] Creating RDMAChannel with id: DataChannel\nI0923 18:30:32.649423 17905 rdma_server_sess.cc:61] received connection: 1/0\nI0923 18:30:32.749850 17906 rdma_adapter.cc:45] RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest would not use shared_cq, the cq is: RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nI0923 18:30:32.749887 17906 rdma_adapter.cc:47] Preparing the resources of RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nI0923 18:30:32.749915 17906 rdma_device.cc:102] Try to get the RDMADevice(mlx5_0) for RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nI0923 18:30:32.749997 17906 rdma_device.cc:108] Warning: RDMADevice (mlx5_0) do not exist\nI0923 18:30:32.750052 17906 rdma_device.cc:67] Creating RDMADevice(mlx5_0)\nI0923 18:30:32.750072 17906 rdma_device.cc:128] The sys params are: cache_line: 64, page_size: 4096\nI0923 18:30:32.750319 17906 rdma_device.cc:205] found 1 device(s)\nI0923 18:30:32.757339 17906 rdma_device.cc:136] Open device (mlx5_0@0x7fd2fdc2c150)\nI0923 18:30:32.757390 17906 rdma_device.cc:150] RDMADevice(mlx5_0) has 1 physical ports\nI0923 18:30:32.757695 17906 rdma_device.cc:177] Info on (1)th port of RDMADevice(mlx5_0) is:\nport_state: 4, max_mtu: 5, activate_mtu: 3, lid: 0, sm_lid: 0, link_layer=2.\nFor more details, please ref to[https://github.com/linux-rdma/rdma-core/blob/486ecb3f12ab17e4b7970a6d5444cd165cec6ee4/libibverbs/verbs.h#L423]\nI0923 18:30:32.757964 17906 rdma_device.cc:74] Try to register an RDMAAdapter in RDMADevice(mlx5_0)\nI0923 18:30:32.758067 17906 rdma_device.cc:86] [Success]: Register RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest into RDMADevice(mlx5_0)\nI0923 18:30:32.758093 17906 rdma_device.cc:119] return RDMADevice(mlx5_0) for RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nI0923 18:30:32.758273 17906 rdma_adapter.cc:97] Creating protection domain(0x7fd2f4000c40) for RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nI0923 18:30:32.758309 17906 rdma_adapter.cc:189] RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest works in active model, i.e., poll it dramatically\nI0923 18:30:32.758383 17906 rdma_device.cc:301] Trying to find the completion queue(CQ), named RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest in RDMADevice(mlx5_0)\nI0923 18:30:32.759755 17906 rdma_device.cc:321] CQ, named RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest is not found in RDMADevice(mlx5_0), create one (0x7fd2f4001470) now!\nI0923 18:30:32.759783 17906 rdma_adapter.cc:132] The CQ@RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest (@0x7fd2f4001470) is privated\nI0923 18:30:32.759793 17906 rdma_adapter.cc:64] Trying to create QPair@RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nW0923 18:30:32.759810 17906 rdma_adapter.cc:76] CQ@RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest would not generate a signal for sqe\nI0923 18:30:32.761607 17906 rdma_adapter.cc:88] [OK]: Successfully create queue_pair(0x7fd2f4001738) for RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nI0923 18:30:32.761658 17906 rdma_adapter.cc:147] Updating the info of RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nW0923 18:30:32.761695 17906 rdma_adapter.cc:157] The actived mtu(3) is less than the config (5), reset it\nI0923 18:30:32.761936 17906 rdma_adapter.cc:221] RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest is trying to connect to the remote (i.e., RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest)\n=====================================================================\n=  Local Adapter ID \t= RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\n=  Remote Adapter ID \t= RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\n=  Local QP number = 0x2f,\tRemote QP number = 0x737c\n=  Local LID \t= 0x0,\t\tRemote LID \t= 0x0\n=  Self GID \t= 00:00:00:00:00:00:00:00:00:00:ff:ff:0c:0c:0c:6f\n=  Remote GID \t= 00:00:00:00:00:00:00:00:00:00:ff:ff:0c:0c:0c:71\n=====================================================================\nI0923 18:30:32.761981 17906 rdma_adapter.cc:340] Modifying QP to INIT, preparing to work\nI0923 18:30:32.762881 17906 rdma_device.cc:277] Querying the attr of QP (0x7fd2f4001738) in RDMADevice(mlx5_0)\nI0923 18:30:32.763243 17906 rdma_adapter.cc:140] Show the qp info in (INIT):\nqp_state: 1, mtu: 1\nI0923 18:30:32.763267 17906 rdma_adapter.cc:289] Modify QP to RTR, ready to receive\nI0923 18:30:32.763274 17906 rdma_adapter.cc:299] Configuure the path_mtu to: 3\nI0923 18:30:32.763280 17906 rdma_adapter.cc:316] Using RoCE\nW0923 18:30:32.763285 17906 rdma_adapter.cc:326] Using traffic class 0\nI0923 18:30:32.764590 17906 rdma_device.cc:277] Querying the attr of QP (0x7fd2f4001738) in RDMADevice(mlx5_0)\nI0923 18:30:32.764794 17906 rdma_adapter.cc:140] Show the qp info in (RTR):\nqp_state: 2, mtu: 3\nI0923 18:30:32.764815 17906 rdma_adapter.cc:265] Modify QP to RTS, ready to send\nI0923 18:30:32.765151 17906 rdma_device.cc:277] Querying the attr of QP (0x7fd2f4001738) in RDMADevice(mlx5_0)\nI0923 18:30:32.765313 17906 rdma_adapter.cc:140] Show the qp info in (RTS):\nqp_state: 3, mtu: 3\nI0923 18:30:32.766242 17907 rdma_buffer.cc:139] Creating buffer : benchmark_buffer\nI0923 18:30:32.766312 17907 rdma_buffer.cc:148] [OK] allocating mem for (benchmark_buffer):  @0x7fd0fffff000, with size: 8388608000\nI0923 18:30:32.766335 17907 rdma_buffer.cc:152] Spliting the buffer benchmark_buffer into 1000 piece(s)\nI0923 18:30:32.770346 17907 rdma_channel.cc:39] Registering buffer (name = benchmark_buffer, size = 8388608000) into RDMAChannel@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nW0923 18:30:32.770390 17907 rdma_device.cc:598] No flags for the mem buf (benchmark_buffer), using the default access mode: (IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE)\nI0923 18:30:35.856914 17907 rdma_device.cc:617] [OK] register mem for (benchmark_buffer):  @0x7fd0fffff000, with size: 8388608000, and flags: 0x7, lkey: 0x6e00, rkey: 0x6e00\nI0923 18:30:35.857262 17907 rdma_device.cc:437] Post receive request to RQ\nI0923 18:30:35.857275 17907 lat_bw_benchmark.cc:425] Register buffer for test@@ConnectionTest\nI0923 18:30:36.065896 17907 rdma_device.cc:383] polled wqe: 1\nI0923 18:30:36.065971 17907 rdma_device.cc:394] Post send request to SQ\nI0923 18:30:36.065997 17907 lat_bw_benchmark.cc:465] Exchange buffer Done\nI0923 18:30:46.022114 17905 lat_bw_benchmark.cc:527] Destroy SchedulingServer\nI0923 18:30:46.022150 17905 rdma_server_sess.cc:21] Destroying the RDMAServerSession of @ConnectionTest\nI0923 18:30:46.022171 17905 rdma_session.cc:55] Destroying the RDMASession of @ConnectionTest\nI0923 18:30:46.022207 17905 rdma_endpoint.cc:10] Destroy RDMAEndPoint\nI0923 18:30:46.268671 17905 rdma_device.cc:633] Successfully deregister mem for benchmark_buffer\nI0923 18:30:46.268790 17905 rdma_channel.cc:63] Buffer(benchmark_buffer) has been removed from RDMAChannel@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\nI0923 18:30:46.268828 17905 rdma_channel.cc:25] Destroying RDMAChannel with id: DataChannel\nI0923 18:30:46.268864 17905 rdma_adapter.cc:12] RDMAAdapter is released\nI0923 18:30:46.269376 17905 rdma_device.cc:60] RDMADevice(mlx5_0) is released\n~/t/F/3/r/build (main)\u003e\n```\n### Run Client\n```shell \n~/t/F/3/r/build (main)\u003e RCL_MAX_VLOG_LEVEL=3 ./lat_bw_benchmark --role=slaver --master=12.12.12.111\nRCL_MAX_VLOG_LEVEL is set to 3, indicating level in VLOG (level) less than 3 would show\nWARNING: Logging before InitGoogleLogging() is written to STDERR\nI0923 18:30:32.639127 56562 rdma_session.cc:36] Creating a new RDMASession for slaver\nI0923 18:30:32.639392 56562 rdma_client_sess.cc:16] Creating a new RDMAClientSession for ConnectionTest\nI0923 18:30:32.639410 56562 lat_bw_benchmark.cc:335] Creating SchedulingClient\nI0923 18:30:32.639456 56562 tcp_connector.cc:118] Preparing socket for RDMAClientSession\nI0923 18:30:32.639480 56562 tcp_connector.cc:123] Creating a socket: 3 for RDMAClientSession\n[INFO] Having modified IP ToS for socket: 3 from 0x0(old) to 0x10(new)\n[INFO] Having modified IP priority for socket: 3 from 6(old) to 4(new)\n[INFO] Having set IP port resue\nI0923 18:30:32.639536 56562 tcp_connector.cc:128] [Done] Preparing socket...\nI0923 18:30:32.639549 56562 rdma_client_sess.cc:31] RDMAClient has prepared everything for Connecting ...\nI0923 18:30:32.639897 56562 rdma_session.cc:156] Handles new connect request by slaver@ConnectionTest\nI0923 18:30:32.639953 56562 rdma_session.cc:159] the remote info of connection request is 12.12.12.111:2020\nI0923 18:30:32.639997 56562 rdma_endpoint.cc:22] Create RDMAEndPoint\nI0923 18:30:32.640374 56562 rdma_endpoint.cc:28] Connection Test OK for TCPConnector@RDMAEndPoint@[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nI0923 18:30:32.640424 56562 rdma_endpoint.cc:90] creating\u0026preparing an rdma channel for DataChannel\nI0923 18:30:32.640468 56562 rdma_adapter.cc:20] Creating a (virtual) adapter for DataChannel\nI0923 18:30:32.640497 56562 rdma_channel.cc:10] Creating RDMAChannel with id: DataChannel\nI0923 18:30:32.640698 56562 rdma_client_sess.cc:54] [Done] Connecting ...\nI0923 18:30:32.741190 56563 rdma_adapter.cc:45] RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest would not use shared_cq, the cq is: RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nI0923 18:30:32.741240 56563 rdma_adapter.cc:47] Preparing the resources of RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nI0923 18:30:32.741262 56563 rdma_device.cc:102] Try to get the RDMADevice(mlx5_0) for RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nI0923 18:30:32.741369 56563 rdma_device.cc:108] Warning: RDMADevice (mlx5_0) do not exist\nI0923 18:30:32.741434 56563 rdma_device.cc:67] Creating RDMADevice(mlx5_0)\nI0923 18:30:32.741454 56563 rdma_device.cc:128] The sys params are: cache_line: 64, page_size: 4096\nI0923 18:30:32.741744 56563 rdma_device.cc:205] found 1 device(s)\nI0923 18:30:32.748852 56563 rdma_device.cc:136] Open device (mlx5_0@0x7f5c2ebc9150)\nI0923 18:30:32.748904 56563 rdma_device.cc:150] RDMADevice(mlx5_0) has 1 physical ports\nI0923 18:30:32.749178 56563 rdma_device.cc:177] Info on (1)th port of RDMADevice(mlx5_0) is:\nport_state: 4, max_mtu: 5, activate_mtu: 3, lid: 0, sm_lid: 0, link_layer=2.\nFor more details, please ref to[https://github.com/linux-rdma/rdma-core/blob/486ecb3f12ab17e4b7970a6d5444cd165cec6ee4/libibverbs/verbs.h#L423]\nI0923 18:30:32.749398 56563 rdma_device.cc:74] Try to register an RDMAAdapter in RDMADevice(mlx5_0)\nI0923 18:30:32.749483 56563 rdma_device.cc:86] [Success]: Register RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest into RDMADevice(mlx5_0)\nI0923 18:30:32.749512 56563 rdma_device.cc:119] return RDMADevice(mlx5_0) for RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nI0923 18:30:32.749676 56563 rdma_adapter.cc:97] Creating protection domain(0x7f5c24000c10) for RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nI0923 18:30:32.749712 56563 rdma_adapter.cc:189] RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest works in active model, i.e., poll it dramatically\nI0923 18:30:32.749773 56563 rdma_device.cc:301] Trying to find the completion queue(CQ), named RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest in RDMADevice(mlx5_0)\nI0923 18:30:32.751170 56563 rdma_device.cc:321] CQ, named RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest is not found in RDMADevice(mlx5_0), create one (0x7f5c24001410) now!\nI0923 18:30:32.751195 56563 rdma_adapter.cc:132] The CQ@RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest (@0x7f5c24001410) is privated\nI0923 18:30:32.751206 56563 rdma_adapter.cc:64] Trying to create QPair@RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nW0923 18:30:32.751224 56563 rdma_adapter.cc:76] CQ@RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest would not generate a signal for sqe\nI0923 18:30:32.753010 56563 rdma_adapter.cc:88] [OK]: Successfully create queue_pair(0x7f5c240016f8) for RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nI0923 18:30:32.753041 56563 rdma_adapter.cc:147] Updating the info of RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nW0923 18:30:32.753072 56563 rdma_adapter.cc:157] The actived mtu(3) is less than the config (5), reset it\nI0923 18:30:32.753273 56563 rdma_adapter.cc:221] RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest is trying to connect to the remote (i.e., RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest)\n=====================================================================\n=  Local Adapter ID \t= RDMAAdapter@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\n=  Remote Adapter ID \t= RDMAAdapter@(DataChannel)[12.12.12.111:2020--\u003e12.12.12.113:60878]@@ConnectionTest\n=  Local QP number = 0x737c,\tRemote QP number = 0x2f\n=  Local LID \t= 0x0,\t\tRemote LID \t= 0x0\n=  Self GID \t= 00:00:00:00:00:00:00:00:00:00:ff:ff:0c:0c:0c:71\n=  Remote GID \t= 00:00:00:00:00:00:00:00:00:00:ff:ff:0c:0c:0c:6f\n=====================================================================\nI0923 18:30:32.753320 56563 rdma_adapter.cc:340] Modifying QP to INIT, preparing to work\nI0923 18:30:32.754289 56563 rdma_device.cc:277] Querying the attr of QP (0x7f5c240016f8) in RDMADevice(mlx5_0)\nI0923 18:30:32.754738 56563 rdma_adapter.cc:140] Show the qp info in (INIT):\nqp_state: 1, mtu: 1\nI0923 18:30:32.754760 56563 rdma_adapter.cc:289] Modify QP to RTR, ready to receive\nI0923 18:30:32.754767 56563 rdma_adapter.cc:299] Configuure the path_mtu to: 3\nI0923 18:30:32.754772 56563 rdma_adapter.cc:316] Using RoCE\nW0923 18:30:32.754778 56563 rdma_adapter.cc:326] Using traffic class 0\nI0923 18:30:32.755736 56563 rdma_device.cc:277] Querying the attr of QP (0x7f5c240016f8) in RDMADevice(mlx5_0)\nI0923 18:30:32.755906 56563 rdma_adapter.cc:140] Show the qp info in (RTR):\nqp_state: 2, mtu: 3\nI0923 18:30:32.755924 56563 rdma_adapter.cc:265] Modify QP to RTS, ready to send\nI0923 18:30:32.756201 56563 rdma_device.cc:277] Querying the attr of QP (0x7f5c240016f8) in RDMADevice(mlx5_0)\nI0923 18:30:32.756348 56563 rdma_adapter.cc:140] Show the qp info in (RTS):\nqp_state: 3, mtu: 3\nI0923 18:30:32.757445 56564 rdma_buffer.cc:139] Creating buffer : benchmark_buffer\nI0923 18:30:32.757514 56564 rdma_buffer.cc:148] [OK] allocating mem for (benchmark_buffer):  @0x7f5a2ffff000, with size: 8388608000\nI0923 18:30:32.757537 56564 rdma_buffer.cc:152] Spliting the buffer benchmark_buffer into 1000 piece(s)\nI0923 18:30:32.766975 56564 rdma_channel.cc:39] Registering buffer (name = benchmark_buffer, size = 8388608000) into RDMAChannel@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nW0923 18:30:32.767031 56564 rdma_device.cc:598] No flags for the mem buf (benchmark_buffer), using the default access mode: (IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE)\nI0923 18:30:36.056674 56564 rdma_device.cc:617] [OK] register mem for (benchmark_buffer):  @0x7f5a2ffff000, with size: 8388608000, and flags: 0x7, lkey: 0x3200, rkey: 0x3200\nI0923 18:30:36.057029 56564 rdma_device.cc:437] Post receive request to RQ\nI0923 18:30:36.057041 56564 lat_bw_benchmark.cc:56] Register buffer for test@slaver@ConnectionTest\nI0923 18:30:36.057101 56564 rdma_device.cc:394] Post send request to SQ\nI0923 18:30:36.057134 56564 rdma_device.cc:383] polled wqe: 1\nI0923 18:30:36.057377 56564 lat_bw_benchmark.cc:97] Exchange buffer Done\nI0923 18:30:37.075444 56564 lat_bw_benchmark.cc:164] Latency for send (1 bytes): 6.577 us/op\nI0923 18:30:37.089924 56564 lat_bw_benchmark.cc:164] Latency for send (2 bytes): 3.095 us/op\nI0923 18:30:37.104373 56564 lat_bw_benchmark.cc:164] Latency for send (4 bytes): 3.126 us/op\nI0923 18:30:37.118826 56564 lat_bw_benchmark.cc:164] Latency for send (8 bytes): 3.133 us/op\nI0923 18:30:37.133219 56564 lat_bw_benchmark.cc:164] Latency for send (16 bytes): 3.079 us/op\nI0923 18:30:37.147665 56564 lat_bw_benchmark.cc:164] Latency for send (32 bytes): 3.141 us/op\nI0923 18:30:37.162684 56564 lat_bw_benchmark.cc:164] Latency for send (64 bytes): 3.708 us/op\nI0923 18:30:37.180263 56564 lat_bw_benchmark.cc:164] Latency for send (128 bytes): 6.273 us/op\nI0923 18:30:37.195271 56564 lat_bw_benchmark.cc:164] Latency for send (256 bytes): 3.699 us/op\nI0923 18:30:37.210341 56564 lat_bw_benchmark.cc:164] Latency for send (512 bytes): 3.758 us/op\nI0923 18:30:37.225495 56564 lat_bw_benchmark.cc:164] Latency for send (1 K bytes): 3.836 us/op\nI0923 18:30:37.240799 56564 lat_bw_benchmark.cc:164] Latency for send (2 K bytes): 4.006 us/op\nI0923 18:30:37.256436 56564 lat_bw_benchmark.cc:164] Latency for send (4 K bytes): 4.332 us/op\nI0923 18:30:37.272644 56564 lat_bw_benchmark.cc:164] Latency for send (8 K bytes): 4.901 us/op\nI0923 18:30:37.292289 56564 lat_bw_benchmark.cc:164] Latency for send (16 K bytes): 8.347 us/op\nI0923 18:30:37.312119 56564 lat_bw_benchmark.cc:164] Latency for send (32 K bytes): 8.523 us/op\nI0923 18:30:37.335009 56564 lat_bw_benchmark.cc:164] Latency for send (64 K bytes): 11.585 us/op\nI0923 18:30:37.364318 56564 lat_bw_benchmark.cc:164] Latency for send (128 K bytes): 17.989 us/op\nI0923 18:30:37.406774 56564 lat_bw_benchmark.cc:164] Latency for send (256 K bytes): 31.142 us/op\nI0923 18:30:37.475414 56564 lat_bw_benchmark.cc:164] Latency for send (512 K bytes): 57.323 us/op\nI0923 18:30:37.595424 56564 lat_bw_benchmark.cc:164] Latency for send (1 M bytes): 108.685 us/op\nI0923 18:30:37.823168 56564 lat_bw_benchmark.cc:164] Latency for send (2 M bytes): 216.417 us/op\nI0923 18:30:38.255426 56564 lat_bw_benchmark.cc:164] Latency for send (4 M bytes): 420.915 us/op\nI0923 18:30:39.100522 56564 lat_bw_benchmark.cc:164] Latency for send (8 M bytes): 833.782 us/op\nI0923 18:30:39.104753 56564 lat_bw_benchmark.cc:200] Latency for write (1 bytes): 3.112 us/op\nI0923 18:30:39.108773 56564 lat_bw_benchmark.cc:200] Latency for write (2 bytes): 2.903 us/op\nI0923 18:30:39.112769 56564 lat_bw_benchmark.cc:200] Latency for write (4 bytes): 2.867 us/op\nI0923 18:30:39.116760 56564 lat_bw_benchmark.cc:200] Latency for write (8 bytes): 2.871 us/op\nI0923 18:30:39.120793 56564 lat_bw_benchmark.cc:200] Latency for write (16 bytes): 2.913 us/op\nI0923 18:30:39.124754 56564 lat_bw_benchmark.cc:200] Latency for write (32 bytes): 2.855 us/op\nI0923 18:30:39.128813 56564 lat_bw_benchmark.cc:200] Latency for write (64 bytes): 2.938 us/op\nI0923 18:30:39.132819 56564 lat_bw_benchmark.cc:200] Latency for write (128 bytes): 2.904 us/op\nI0923 18:30:39.136874 56564 lat_bw_benchmark.cc:200] Latency for write (256 bytes): 2.928 us/op\nI0923 18:30:39.141011 56564 lat_bw_benchmark.cc:200] Latency for write (512 bytes): 3.016 us/op\nI0923 18:30:39.145241 56564 lat_bw_benchmark.cc:200] Latency for write (1 K bytes): 3.112 us/op\nI0923 18:30:39.149686 56564 lat_bw_benchmark.cc:200] Latency for write (2 K bytes): 3.323 us/op\nI0923 18:30:39.154464 56564 lat_bw_benchmark.cc:200] Latency for write (4 K bytes): 3.677 us/op\nI0923 18:30:39.159955 56564 lat_bw_benchmark.cc:200] Latency for write (8 K bytes): 4.374 us/op\nI0923 18:30:39.166767 56564 lat_bw_benchmark.cc:200] Latency for write (16 K bytes): 5.691 us/op\nI0923 18:30:39.176163 56564 lat_bw_benchmark.cc:200] Latency for write (32 K bytes): 8.274 us/op\nI0923 18:30:39.188625 56564 lat_bw_benchmark.cc:200] Latency for write (64 K bytes): 11.359 us/op\nI0923 18:30:39.207278 56564 lat_bw_benchmark.cc:200] Latency for write (128 K bytes): 17.533 us/op\nI0923 18:30:39.238997 56564 lat_bw_benchmark.cc:200] Latency for write (256 K bytes): 30.586 us/op\nI0923 18:30:39.299543 56564 lat_bw_benchmark.cc:200] Latency for write (512 K bytes): 59.426 us/op\nI0923 18:30:39.406946 56564 lat_bw_benchmark.cc:200] Latency for write (1 M bytes): 106.286 us/op\nI0923 18:30:39.619195 56564 lat_bw_benchmark.cc:200] Latency for write (2 M bytes): 211.122 us/op\nI0923 18:30:40.041168 56564 lat_bw_benchmark.cc:200] Latency for write (4 M bytes): 420.851 us/op\nI0923 18:30:40.881762 56564 lat_bw_benchmark.cc:200] Latency for write (8 M bytes): 839.442 us/op\nI0923 18:30:40.890528 56564 lat_bw_benchmark.cc:327] Latency for read (1 bytes): 7.647 us/op\nI0923 18:30:40.894783 56564 lat_bw_benchmark.cc:327] Latency for read (2 bytes): 3.12 us/op\nI0923 18:30:40.899071 56564 lat_bw_benchmark.cc:327] Latency for read (4 bytes): 3.165 us/op\nI0923 18:30:40.903365 56564 lat_bw_benchmark.cc:327] Latency for read (8 bytes): 3.168 us/op\nI0923 18:30:40.907655 56564 lat_bw_benchmark.cc:327] Latency for read (16 bytes): 3.169 us/op\nI0923 18:30:40.911936 56564 lat_bw_benchmark.cc:327] Latency for read (32 bytes): 3.16 us/op\nI0923 18:30:40.916200 56564 lat_bw_benchmark.cc:327] Latency for read (64 bytes): 3.142 us/op\nI0923 18:30:40.920523 56564 lat_bw_benchmark.cc:327] Latency for read (128 bytes): 3.199 us/op\nI0923 18:30:40.924831 56564 lat_bw_benchmark.cc:327] Latency for read (256 bytes): 3.199 us/op\nI0923 18:30:40.929280 56564 lat_bw_benchmark.cc:327] Latency for read (512 bytes): 3.321 us/op\nI0923 18:30:40.933799 56564 lat_bw_benchmark.cc:327] Latency for read (1 K bytes): 3.397 us/op\nI0923 18:30:40.938514 56564 lat_bw_benchmark.cc:327] Latency for read (2 K bytes): 3.608 us/op\nI0923 18:30:40.943511 56564 lat_bw_benchmark.cc:327] Latency for read (4 K bytes): 3.873 us/op\nI0923 18:30:40.948794 56564 lat_bw_benchmark.cc:327] Latency for read (8 K bytes): 4.173 us/op\nI0923 18:30:40.954860 56564 lat_bw_benchmark.cc:327] Latency for read (16 K bytes): 4.937 us/op\nI0923 18:30:40.962416 56564 lat_bw_benchmark.cc:327] Latency for read (32 K bytes): 6.421 us/op\nI0923 18:30:40.973601 56564 lat_bw_benchmark.cc:327] Latency for read (64 K bytes): 10.053 us/op\nI0923 18:30:40.991923 56564 lat_bw_benchmark.cc:327] Latency for read (128 K bytes): 17.246 us/op\nI0923 18:30:41.024709 56564 lat_bw_benchmark.cc:327] Latency for read (256 K bytes): 31.651 us/op\nI0923 18:30:41.086304 56564 lat_bw_benchmark.cc:327] Latency for read (512 K bytes): 60.474 us/op\nI0923 18:30:41.208410 56564 lat_bw_benchmark.cc:327] Latency for read (1 M bytes): 120.985 us/op\nI0923 18:30:41.448765 56564 lat_bw_benchmark.cc:327] Latency for read (2 M bytes): 239.237 us/op\nI0923 18:30:41.927474 56564 lat_bw_benchmark.cc:327] Latency for read (4 M bytes): 477.588 us/op\nI0923 18:30:42.865584 56564 lat_bw_benchmark.cc:327] Latency for read (8 M bytes): 936.959 us/op\nI0923 18:30:42.865626 56564 lat_bw_benchmark.cc:370] Using round-robin data buffer for BW test\nI0923 18:30:42.867002 56564 lat_bw_benchmark.cc:246] BW for write (1 bytes): 0.030303 Gbps\nI0923 18:30:42.868515 56564 lat_bw_benchmark.cc:246] BW for write (2 bytes): 0.040404 Gbps\nI0923 18:30:42.870071 56564 lat_bw_benchmark.cc:246] BW for write (4 bytes): 0.0742459 Gbps\nI0923 18:30:42.871392 56564 lat_bw_benchmark.cc:246] BW for write (8 bytes): 0.323232 Gbps\nI0923 18:30:42.872851 56564 lat_bw_benchmark.cc:246] BW for write (16 bytes): 0.378698 Gbps\nI0923 18:30:42.874380 56564 lat_bw_benchmark.cc:246] BW for write (32 bytes): 0.633663 Gbps\nI0923 18:30:42.875881 56564 lat_bw_benchmark.cc:246] BW for write (64 bytes): 1.35092 Gbps\nI0923 18:30:42.877311 56564 lat_bw_benchmark.cc:246] BW for write (128 bytes): 3.55556 Gbps\nI0923 18:30:42.878600 56564 lat_bw_benchmark.cc:246] BW for write (256 bytes): 11.0108 Gbps\nI0923 18:30:42.880005 56564 lat_bw_benchmark.cc:246] BW for write (512 bytes): 14.2718 Gbps\nI0923 18:30:42.881321 56564 lat_bw_benchmark.cc:246] BW for write (1 K bytes): 43.3439 Gbps\nI0923 18:30:42.882778 56564 lat_bw_benchmark.cc:246] BW for write (2 K bytes): 49.0539 Gbps\nI0923 18:30:42.884287 56564 lat_bw_benchmark.cc:246] BW for write (4 K bytes): 85.7801 Gbps\nI0923 18:30:42.886178 56564 lat_bw_benchmark.cc:246] BW for write (8 K bytes): 87.265 Gbps\nI0923 18:30:42.888906 56564 lat_bw_benchmark.cc:246] BW for write (16 K bytes): 81.8688 Gbps\nI0923 18:30:42.893222 56564 lat_bw_benchmark.cc:246] BW for write (32 K bytes): 82.0739 Gbps\nI0923 18:30:42.900738 56564 lat_bw_benchmark.cc:246] BW for write (64 K bytes): 82.2026 Gbps\nI0923 18:30:42.925539 56564 lat_bw_benchmark.cc:246] BW for write (128 K bytes): 44.311 Gbps\nI0923 18:30:42.952082 56564 lat_bw_benchmark.cc:246] BW for write (256 K bytes): 82.4903 Gbps\nI0923 18:30:43.004007 56564 lat_bw_benchmark.cc:246] BW for write (512 K bytes): 82.5667 Gbps\nI0923 18:30:43.106720 56564 lat_bw_benchmark.cc:246] BW for write (1 M bytes): 82.5707 Gbps\nI0923 18:30:43.310956 56564 lat_bw_benchmark.cc:246] BW for write (2 M bytes): 82.5996 Gbps\nI0923 18:30:43.718415 56564 lat_bw_benchmark.cc:246] BW for write (4 M bytes): 82.5843 Gbps\nI0923 18:30:44.533110 56564 lat_bw_benchmark.cc:246] BW for write (8 M bytes): 82.4888 Gbps\nI0923 18:30:44.533143 56564 lat_bw_benchmark.cc:376] Using one data buffer for BW test\nI0923 18:30:44.534597 56564 lat_bw_benchmark.cc:292] BW for write (1 bytes): 0.0235294 Gbps\nI0923 18:30:44.535938 56564 lat_bw_benchmark.cc:292] BW for write (2 bytes): 0.0772947 Gbps\nI0923 18:30:44.537402 56564 lat_bw_benchmark.cc:292] BW for write (4 bytes): 0.0943953 Gbps\nI0923 18:30:44.538902 56564 lat_bw_benchmark.cc:292] BW for write (8 bytes): 0.170213 Gbps\nI0923 18:30:44.540205 56564 lat_bw_benchmark.cc:292] BW for write (16 bytes): 0.719101 Gbps\nI0923 18:30:44.541656 56564 lat_bw_benchmark.cc:292] BW for write (32 bytes): 0.780488 Gbps\nI0923 18:30:44.543131 56564 lat_bw_benchmark.cc:292] BW for write (64 bytes): 1.50147 Gbps\nI0923 18:30:44.544441 56564 lat_bw_benchmark.cc:292] BW for write (128 bytes): 5.88506 Gbps\nI0923 18:30:44.545745 56564 lat_bw_benchmark.cc:292] BW for write (256 bytes): 11.7029 Gbps\nI0923 18:30:44.547050 56564 lat_bw_benchmark.cc:292] BW for write (512 bytes): 23.4057 Gbps\nI0923 18:30:44.548503 56564 lat_bw_benchmark.cc:292] BW for write (1 K bytes): 24.9756 Gbps\nI0923 18:30:44.549894 56564 lat_bw_benchmark.cc:292] BW for write (2 K bytes): 64.7589 Gbps\nI0923 18:30:44.551398 56564 lat_bw_benchmark.cc:292] BW for write (4 K bytes): 89.7753 Gbps\nI0923 18:30:44.553236 56564 lat_bw_benchmark.cc:292] BW for write (8 K bytes): 90.896 Gbps\nI0923 18:30:44.555785 56564 lat_bw_benchmark.cc:292] BW for write (16 K bytes): 91.7871 Gbps\nI0923 18:30:44.559749 56564 lat_bw_benchmark.cc:292] BW for write (32 K bytes): 92.4018 Gbps\nI0923 18:30:44.566542 56564 lat_bw_benchmark.cc:292] BW for write (64 K bytes): 92.4018 Gbps\nI0923 18:30:44.578979 56564 lat_bw_benchmark.cc:292] BW for write (128 K bytes): 92.5241 Gbps\nI0923 18:30:44.602779 56564 lat_bw_benchmark.cc:292] BW for write (256 K bytes): 92.5445 Gbps\nI0923 18:30:44.649219 56564 lat_bw_benchmark.cc:292] BW for write (512 K bytes): 92.5609 Gbps\nI0923 18:30:44.740983 56564 lat_bw_benchmark.cc:292] BW for write (1 M bytes): 92.5537 Gbps\nI0923 18:30:44.923388 56564 lat_bw_benchmark.cc:292] BW for write (2 M bytes): 92.5481 Gbps\nI0923 18:30:45.287003 56564 lat_bw_benchmark.cc:292] BW for write (4 M bytes): 92.5655 Gbps\nI0923 18:30:46.013114 56564 lat_bw_benchmark.cc:292] BW for write (8 M bytes): 92.5673 Gbps\nI0923 18:30:46.013352 56562 lat_bw_benchmark.cc:339] Destroying SchedulingClient\nI0923 18:30:46.013383 56562 rdma_client_sess.cc:22] Destroying the RDMAClientSession of ConnectionTest\nI0923 18:30:46.013396 56562 rdma_session.cc:55] Destroying the RDMASession of slaver@ConnectionTest\nI0923 18:30:46.013443 56562 rdma_endpoint.cc:9] Destroy RDMAEndPoint\nI0923 18:30:46.296687 56562 rdma_device.cc:633] Successfully deregister mem for benchmark_buffer\nI0923 18:30:46.296794 56562 rdma_channel.cc:63] Buffer(benchmark_buffer) has been removed from RDMAChannel@(DataChannel)[12.12.12.113:60878--\u003e12.12.12.111:2020]@slaver@ConnectionTest\nI0923 18:30:46.296826 56562 rdma_channel.cc:25] Destroying RDMAChannel with id: DataChannel\nI0923 18:30:46.296864 56562 rdma_adapter.cc:12] RDMAAdapter is released\nI0923 18:30:46.297307 56562 rdma_device.cc:60] RDMADevice(mlx5_0) is released\n~/t/F/3/r/build (main)\u003e\n```\n# Updates\n- The initialization version is uploaded.\n\n\n\n\n# References\nBelows are some useful references that may help you have a better understanding on RDMA\n - [RDMA open source library](https://libs.garden/cpp/search?q=rdma\u0026page=2)\n - [analyze RDMA code](https://www.cnblogs.com/vlhn/p/7997457.html)\n - [analyze RDMA design](https://www.zhihu.com/column/c_1231181516811390976)\n - [RDMA Wheel](https://github.com/jcf94/RDMA-wheel)\n - [introduction-to-programming-infiniband](https://insujang.github.io/2020-02-09/introduction-to-programming-infiniband/)\n - [perf usage](http://www.brendangregg.com/perf.html)\n - [Optimizing All_to_All primitives for Infiniband networks](https://wenku.baidu.com/view/4d46db77856a561252d36ffa.html)\n - [VMA performance optimization](https://community.mellanox.com/s/article/vma-performance-tuning-guide)","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnewplan%2Frdma_comm_core","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnewplan%2Frdma_comm_core","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnewplan%2Frdma_comm_core/lists"}