{"id":18745862,"url":"https://github.com/emersonford/thesis","last_synced_at":"2025-11-23T15:30:15.560Z","repository":{"id":111835985,"uuid":"294033577","full_name":"emersonford/thesis","owner":"emersonford","description":"Repository for my undergraduate honors CS thesis: Evaluating the use of RDMA in High Performance Containers.","archived":false,"fork":false,"pushed_at":"2022-05-14T00:30:35.000Z","size":24450,"stargazers_count":2,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"master","last_synced_at":"2024-12-28T20:26:53.676Z","etag":null,"topics":["containers","rdma"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/emersonford.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-09T07:16:38.000Z","updated_at":"2024-01-30T04:18:39.000Z","dependencies_parsed_at":"2023-06-04T07:45:13.200Z","dependency_job_id":null,"html_url":"https://github.com/emersonford/thesis","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emersonford%2Fthesis","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emersonford%2Fthesis/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emersonford%2Fthesis/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/emersonford%2Fthesis/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/emersonford","download_url":"https://codeload.github.com/emersonford/thesis/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":239627235,"owners_count":19670844,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["containers","rdma"],"created_at":"2024-11-07T16:19:50.377Z","updated_at":"2025-11-23T15:30:15.495Z","avatar_url":"https://github.com/emersonford.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"# Explaining and Evaluating the Use of RDMA in High Performance Containers\nHonors Computer Science Bachelor's Thesis by Emerson Ford at the University of Utah.\n\n## Folders\n1. `ansible` an ansible playbook to setup hosts for testing various RDMA-in-container solutions.\n2. `bg_presentation` background presentation on the topic of this thesis\n3. `data` raw data gathered for each RDMA-in-container solution test\n4. `Freeflow` submodule to a fork of Freeflow, which required a few alterations to get working and includes some QoL changes to make testing faster\n5. `paper` actual document in LaTeX for this thesis\n\n## Reproducing Data\nThis assumes you're using [Cloudlab](https://www.cloudlab.us/) and have SSH keys configured on your Cloudlab account. You should also have `ansible-playbook` and Python 3.10 installed.\n\n### SoftRoCE Tests\n1. Provision a Cloudlab experiment with the [roce-cluster](https://www.cloudlab.us/show-profile.php?uuid=fbcf91c3-93ba-11ec-9467-e4434b2381fc) profile.\n    * Change \"Node type to use\" to `d6515`.\n2. After the hosts have booted, upgrade them to Ubuntu 21.10. This is pretty much just running `sudo do-release-upgrade` on the hosts. \n3. Change the two hostnames of the `no_mlnx_ofed` group to your Cloudlab hostnames in `ansible/hosts.yaml`. Change `ansible_user` under `vars` to your Cloudlab username.\n4. Run `ansible-playbook --ssh-common-args \"-o StrictHostKeyChecking=no\" --inventory-file=\"./hosts.yaml\" --limit no_mlnx_ofed site.yml` while in the `ansible` directory.\n5. Run the commands listed in `data/softroce_*/metadata*` while in the `test_scripts` directory. Take care to replace the `--host1` and `--host2` flags to match your Cloudlab hostnames, and `--user` to match your Cloudlab username. The first argument (`/opt/homebrew/.../Python`) should be replaced with the path to your Python 3.10 binary. \n    * Run the host (i.e. non-SoftRoCE) versions of the test first.\n    * Then, for running the SoftRoCE version of `run_basic_tests` and `run_cpu_tests`, a SoftRoCE NIC must be manually configured before running the tests. SSH into both hosts and run `sudo modprobe -rv mlx5_ib \u0026\u0026 sudo reboot now`. After they reboot, run `sudo rdma link | grep rxe0 || sudo rdma link add rxe0 type rxe netdev enp65s0f0np0 \u0026\u0026 sudo devlink dev param set pci/0000:41:00.0 name enable_roce value false cmode driverinit \u0026\u0026 sudo devlink dev reload pci/0000:41:00.0`. Then run the test commands.\n6. The data should appear in `data/raw`. You can generate graphs based on the data you just produced by setting `MODE = \"softroce\"` and rerunning all cells in the `*.ipynb` Jupyter notebooks (run `jupyter-notebook` while in the `data` dir).\n\n#### Quirks to Watch Out For\n1. Ubuntu 21.10 / Linux Kernel \u003e5.13 is required to avoid certain kernel panics when using SoftRoCE.\n\n### Shared HCA Tests\n1. Provision a Cloudlab experiment with the [roce-cluster](https://www.cloudlab.us/show-profile.php?uuid=fbcf91c3-93ba-11ec-9467-e4434b2381fc) profile.\n    * Change \"Node type to use\" to `d6515`.\n2. Change the two hostnames of the `connectx5` group to your Cloudlab hostnames in `ansible/hosts.yaml`. Change `ansible_user` under `vars:` to your Cloudlab username.\n3. Run `ansible-playbook --ssh-common-args \"-o StrictHostKeyChecking=no\" --inventory-file=\"./hosts.yaml\" --limit connectx5 site.yml` while in the `ansible` directory.\n4. Run the commands listed in `data/shared_hca_*/metadata*` while in the `test_scripts` directory. Take care to replace the `--host1` and `--host2` flags to match your Cloudlab hostnames, and `--user` to match your Cloudlab username. The first argument (`/opt/homebrew/.../Python`) should be replaced with the path to your Python 3.10 binary. \n    * For running the Shared HCA version of `run_basic_tests` and `run_cpu_tests`, a Docker macvlan network must first be created with `docker network ls | grep mynet || docker network create -d macvlan --subnet=192.168.1.0/24 -o parent=ens3f0 -o macvlan_mode=private mynet`.\n6. The data should appear in `data/raw`. You can generate graphs based on the data you just produced by setting `MODE = \"shared_hca\"` and rerunning all cells in the `*.ipynb` Jupyter notebooks (run `jupyter-notebook` while in the `data` dir).\n\n#### Quirks to Watch Out For\n1. The RDMA GID table is namespaced inside of the container, thus the majority of GID entries are `0000:0000:0000:0000:0000:0000:0000:0000`, and the ones namespaced into the container's namespace are populated. Despite this, `ib_[read|write|send]_[bw|lat]` do not select the proper GID and will error with `Failed to modify QP XXXX to RTR` and `Unable to Connect the HCA's through the link`. You can force `ib_[read|write|send]_[bw|lat]` to use the correct GID entry with the `-x` flag. See `/sys/class/infiniband/*/ports/*/gids` for GID entry values and `/sys/class/infiniband/*/ports/*/gid_attrs/types` for the corresponding type (RoCE v1, RoCE v2, etc).\n    * You can use `rdma_cm` queue pairs to avoid this with the `-R` flag. However, using RDMA connection manager queue pairs results in 100% CPU utilization on the `ib_[read|write|send]_[bw|lat]` server (which should have around a 0% CPU util for read/write operations), thus their use can result in incorrect CPU usage readings.\n\n### SRIOV Tests\n1. Provision a Cloudlab experiment with the [roce-cluster](https://www.cloudlab.us/show-profile.php?uuid=fbcf91c3-93ba-11ec-9467-e4434b2381fc) profile.\n    * Change \"Node type to use\" to `d6515`.\n2. Change the two hostnames of the `connectx5` group to your Cloudlab hostnames in `ansible/hosts.yaml`. Change `ansible_user` under `vars:` to your Cloudlab username.\n3. Run `ansible-playbook --ssh-common-args \"-o StrictHostKeyChecking=no\" --inventory-file=\"./hosts.yaml\" --limit connectx5 site.yml` while in the `ansible` directory.\n4. Run the commands listed in `data/sriov_basic_tests/metadata_host` and `data/sriov_cpu_tests/metadata_host` while in the `test_scripts` directory. Take care to replace the `--host1` and `--host2` flags to match your Cloudlab hostnames, and `--user` to match your Cloudlab username. The first argument (`/opt/homebrew/.../Python`) should be replaced with the path to your Python 3.10 binary. \n5. Run `ansible-playbook --ssh-common-args \"-o StrictHostKeyChecking=no\" --inventory-file=\"./hosts.yaml\" --limit connectx5 --tags sriov site.yml` while in the `ansible` directory. This will provision the first SRIOV virtual function on both hosts.\n6. Run the commands listed in `data/sriov_basic_tests/metadata_sriov` and `data/sriov_cpu_tests/metadata_sriov` while in the `test_scripts` directory. Take care to replace the `--host1` and `--host2` flags to match your Cloudlab hostnames, and `--user` to match your Cloudlab username. The first argument (`/opt/homebrew/.../Python`) should be replaced with the path to your Python 3.10 binary. \n7. Run the commands listed in `data/sriov_multi_dev/metadata` while in the `test_scripts` directory. Take care to replace the `--host1` and `--host2` flags to match your Cloudlab hostnames, and `--user` to match your Cloudlab username. The first argument (`/opt/homebrew/.../Python`) should be replaced with the path to your Python 3.10 binary. \n8. The data should appear in `data/raw`. You can generate graphs based on the data you just produced by setting `MODE = \"sriov\"` and rerunning all cells in the `*.ipynb` Jupyter notebooks (run `jupyter-notebook` while in the `data` dir).\n\n#### Quirks to Watch Out For\n1. SRIOV virtual function instantiation is really finicky. Sometimes it behaves and sometimes it doesn't. If your `basic_tests` or `cpu_tests` don't work, reboot the host and rerun `ansible-playbook --ssh-common-args \"-o StrictHostKeyChecking=no\" --inventory-file=\"./hosts.yaml\" --limit connectx5 --tags sriov site.yml`. Then rerun your tests.\n    * The `multi_sriov_tests.py` script tries to handle the finickiness of SRIOV virtual functions, but after \u003e20 times to get them to cooperate, it will fail the test.\n\n### Freeflow Tests\n1. Provision a Cloudlab experiment with the [roce-cluster](https://www.cloudlab.us/show-profile.php?uuid=fbcf91c3-93ba-11ec-9467-e4434b2381fc) profile.\n    * Change \"Node type to use\" to `c6220`.\n2. Change the two hostnames of the `connectx3` group to your Cloudlab hostnames in `ansible/hosts.yaml`. Change `ansible_user` under `vars:` to your Cloudlab username.\n3. Run `ansible-playbook --ssh-common-args \"-o StrictHostKeyChecking=no\" --inventory-file=\"./hosts.yaml\" --limit connectx3 site.yml` while in the `ansible` directory.\n4. Run the commands listed in `data/freeflow_*/metadata*` while in the `test_scripts` directory. Take care to replace the `--host1` and `--host2` flags to match your Cloudlab hostnames, and `--user` to match your Cloudlab username. The first argument (`/opt/homebrew/.../Python`) should be replaced with the path to your Python 3.10 binary. \n5. The data should appear in `data/raw`. You can generate graphs based on the data you just produced by setting `MODE = \"freeflow\"` and rerunning all cells in the `*.ipynb` Jupyter notebooks (run `jupyter-notebook` while in the `data` dir).\n\n#### Quirks to Watch Out For\n1. RDMA's rkey generation is deterministic (see the [ReDMArk paper](https://www.usenix.org/system/files/sec21-rothenberger.pdf)), particularly on mlx4 NICs. Freeflow assumes unique rkeys per host as part of its rkey mapping scheme, which breaks with this deterministic generation. I added a patch to my fork of Freeflow to circumvent this, but if you run into `Failed status 10: wr_id 0 syndrom 0x88` errors, this is likely why. \n2. Freeflow expects page-aligned memory, hence the use of `LD_PRELOAD=./align_malloc.so`. \n3. Freeflow only supports mlx4 driver NICs, so you must use ConnectX-3 NICs.\n4. Freeflow provides a \"no-fastpath\" mode. However, this mode is prone to deadlocks at specific RDMA packet sizes and with more than 2 clients.\n\n### Mellanox ASAP2 Direct Tests\n1. Provision a Cloudlab experiment with the [roce-cluster](https://www.cloudlab.us/show-profile.php?uuid=fbcf91c3-93ba-11ec-9467-e4434b2381fc) profile.\n    * Change \"Node type to use\" to `c6525-100g`.\n2. Change the two hostnames of the `connectx5` group to your Cloudlab hostnames in `ansible/hosts.yaml`. Change `ansible_user` under `vars:` to your Cloudlab username.\n3. Run `ansible-playbook --ssh-common-args \"-o StrictHostKeyChecking=no\" --inventory-file=\"./hosts.yaml\" --limit connectx5 --tags sriov,asap2_direct site.yml` while in the `ansible` directory. Reboot both hosts through the Cloudlab UI after installing MLNX OFED. Then rerun the `ansible-playbook` command to completion.\n4. Run the commands listed in `data/asap2_direct_basic_tests/metadata_host` and `data/asap2_direct_cpu_tests/metadata_host` while in the `test_scripts` directory. Take care to replace the `--host1` and `--host2` flags to match your Cloudlab hostnames, and `--user` to match your Cloudlab username. The first argument (`/opt/homebrew/.../Python`) should be replaced with the path to your Python 3.10 binary. \n5. Run `ansible-playbook --ssh-common-args \"-o StrictHostKeyChecking=no\" --inventory-file=\"./hosts.yaml\" --limit connectx5 --tags sriov,asap2_direct -e \"configure_sriov_ifs=false\" site.yml` while in the `ansible` directory. This will provision the first SRIOV virtual function on both hosts and configure ASAP2 Direct.\n6. Run the commands listed in `data/asap2_direct_basic_tests/metadata_sriov_ovs` and `data/asap2_direct_cpu_tests/metadata_sriov_ovs` while in the `test_scripts` directory. Take care to replace the `--host1` and `--host2` flags to match your Cloudlab hostnames, and `--user` to match your Cloudlab username. The first argument (`/opt/homebrew/.../Python`) should be replaced with the path to your Python 3.10 binary. \n7. Run `ansible-playbook --ssh-common-args \"-o StrictHostKeyChecking=no\" --inventory-file=\"./hosts.yaml\" --limit connectx5 --tags sriov,asap2_direct -e \"NUM_OF_VFS=32\" -e \"configure_sriov_ifs=false\" -e \"cleanup_old_state=true\" site.yml` while in the `ansible` directory. This will provision multiple VFs for the multi dev test.\n7. Run the commands listed in `data/asap2_direct_multi_dev/metadata` while in the `test_scripts` directory. Take care to replace the `--host1` and `--host2` flags to match your Cloudlab hostnames, and `--user` to match your Cloudlab username. The first argument (`/opt/homebrew/.../Python`) should be replaced with the path to your Python 3.10 binary. \n8. The data should appear in `data/raw`. You can generate graphs based on the data you just produced by setting `MODE = \"asap2_direct\"` and rerunning all cells in the `*.ipynb` Jupyter notebooks (run `jupyter-notebook` while in the `data` dir).\n\n#### Quirks to Watch Out For\n1. When using `switchdev`, there's both an interface for the SRIOV NIC itself and a \"representor netdevice\" (see [these slides](https://legacy.netdevconf.info/1.2/session.html?or-gerlitz)). Sometimes the names get messed up on these and you have to reboot the host or mess around with udev rules. \n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femersonford%2Fthesis","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Femersonford%2Fthesis","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Femersonford%2Fthesis/lists"}