{"id":23657770,"url":"https://github.com/aliireza/packetmill","last_synced_at":"2025-08-23T12:33:31.326Z","repository":{"id":78240615,"uuid":"317246969","full_name":"aliireza/packetmill","owner":"aliireza","description":"PacketMill: Toward per-core 100-Gbps Networking","archived":false,"fork":false,"pushed_at":"2022-02-01T12:47:06.000Z","size":9575,"stargazers_count":62,"open_issues_count":0,"forks_count":9,"subscribers_count":5,"default_branch":"master","last_synced_at":"2025-05-20T10:12:16.097Z","etag":null,"topics":["dpdk","fastclick","llvm-pass","packet-processing","packetmill","x-change"],"latest_commit_sha":null,"homepage":"http://packetmill.github.io/","language":"Gnuplot","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/aliireza.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":"CITATION.cff","codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2020-11-30T14:21:54.000Z","updated_at":"2025-04-17T03:54:16.000Z","dependencies_parsed_at":"2023-05-23T14:30:55.738Z","dependency_job_id":null,"html_url":"https://github.com/aliireza/packetmill","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/aliireza/packetmill","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aliireza%2Fpacketmill","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aliireza%2Fpacketmill/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aliireza%2Fpacketmill/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aliireza%2Fpacketmill/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/aliireza","download_url":"https://codeload.github.com/aliireza/packetmill/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/aliireza%2Fpacketmill/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":271747023,"owners_count":24813604,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-08-23T02:00:09.327Z","response_time":69,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["dpdk","fastclick","llvm-pass","packet-processing","packetmill","x-change"],"created_at":"2024-12-28T21:58:01.650Z","updated_at":"2025-08-23T12:33:31.317Z","avatar_url":"https://github.com/aliireza.png","language":"Gnuplot","funding_links":[],"categories":[],"sub_categories":[],"readme":"# PacketMill: Toward per-core 100-Gbps Networking\n\nPacketMill is a system for optimizing software packet processing, which (i) introduces a new model ([X-Change][x-change-repo]) to efficiently manage packet metadata and (ii) employs code-optimization techniques to better utilize commodity hardware. PacketMill grinds the whole packet processing stack, from the high-level network function configuration file to the low-level userspace network (specifically Data Plane Development Kit or [DPDK][dpdk-page]) drivers, to mitigate inefficiencies and produce a customized binary for a given network function.\n\n\u003cp align=\"center\"\u003e\n\u003cimg src=\"PacketMill-logo.png\"  alt=\"PacketMill\" width=\"40%\"\u003e\n\u003c/p\u003e\n\u003cp align=\"center\"\u003e\n\u003csub\u003eSome of the symbols used in this image are modified versions of vectors published by \u003ca href=\"https://www.freevector.com/\"\u003ewww.freevector.com\u003c/a\u003e.\u003c/sub\u003e\n\u003c/p\u003e\n\n\nPacketMill is composed of three main components:\n\n1. **X-Change:** developed as an Application Programming Interface (API) within DPDK, which realizes customized\nbuffers when using DPDK, rather than relying on the generic ***rte_mbuf*** structure;\n2. **Source-code modifications:** implemented on top of a resurrected \u0026 modified ***click-devirtualize***, exploiting\nthe information defining a NF (i.e., Click configuration file) to mitigate virtual calls, improve constant propagation \u0026 constant folding; and\n1. **IR-based modifications:** implemented as LLVM optimization passes applied to the complete program’s IR bitcode\nas extracted from Link-Time Optimization (LTO), which reoders the commonly used data structures (i.e., ***Packet*** class in FastClick).\n\nFor more information, please refer to PacketMill's [paper][packetmill-paper]. You can also check out our submitted [extended abstract][packetmill-extended-abstract] to [ASPLOS'21][asplos21-page].\n\n## Applicability\n\nWe have developed/tested PacketMill for/on [FastClick][fastclick-repo], but our techniques could be adapted to other packet processing frameworks.\n\n### X-Change (Modified DPDK)\n\nWe modified the `MLX5` PMD used by Mellanox NICs in DPDK. However, X-Change is applicable to other drivers, as other (e.g., Intel) drivers are implemented similarly and have the same inefficiencies. Although we have only tested X-Change with FastClick, other DPDK-based packet processing frameworks (e.g., BESS and VPP) could equally benefit from X-Change, as our proposed model only modifies DPDK userspace drivers.\n\n\n### Code Optimizations\n\nWe implemented our source-code optimizations on top of [click-devirtualize][devirtualize-paper], but it is possible to apply the same optimization to other packet processing frameworks such as BESS and VPP.\n\n\n## What is included here?\n\nThis repository contains information/source code to use PacketMill and reproduce some of the results presented in our ASPLOS'21 [paper][packetmill-paper].\n\nOur paper' results are also available at `paper-results/`. \n\n\n## Experiments\n\nThe experiments are located at `experiments/`. The folder has a `Makefile` and `README.md` that can be used to run the experiments.\n\n**Note: Before running the experiments, you need to prepare your testbed according to the following guidelines.**\n\n\n## Testbed\n\nOur experiments mainly requires `npf`, `X-Change`, `FastClick`, `LLVM toolchain`, `perf`, and `pmu-tools`. There is a simple bash script (`setup_repo.sh`) that could help you to clone/compile different repositories, but you should mainly rely on this `README.md`. \n\n### Network Performance Framework (NPF) Tool\n\nYou can install `npf` via the following command:\n\n```bash\npython3 -m pip install --user npf\n```\n\n**Do not forget to add `export PATH=$PATH:~/.local/bin` to `~/.bashrc` or `~/.zshrc`. Otherwise, you cannot run `npf-compare` and `npf-run` commands.** \n\nNPF will look for `cluster/` and `repo/` in your current working/testie directory. We have included the required `repo` for our experiments and a sample `cluster` template, available at `experiment/`. To setup your cluster, please check the [guidelines][npf-setup] for our previous paper. Additionally, you can check the [NPF README][npf-readme] file.\n\n### X-Change (Modified DPDK) and Normal DPDK\n\nTo build X-Change with clang (LTO), you can run the following commands:\n\n```bash\ngit clone https://github.com/tbarbette/xchange.git\ncd xchange\nmake install T=x86_64-native-linux-clanglto\n```\n\nAfter building X-Change, you have to define `XCHG_SDK` and `XCHG_TARGET`. To do so, run something similar to:\n\n```bash\nexport XCHG_SDK=/home/alireza/packetmill/xchange/\nexport XCHG_TARGET=x86_64-native-linux-clanglto\n```\n\nWe also use normal DPDK v20.02 in some scenarios. To build it, you can run the following commands:\n\n```bash\ngit clone https://github.com/tbarbette/xchange.git dpdk\ncd dpdk\ngit checkout v20.02\nmake install T=x86_64-native-linux-gcc\nmake install T=x86_64-native-linux-clang\n```\n\nAfter building DPDK, you have to define `RTE_SDK` and `RTE_TARGET`. To do so, run run something similar to:\n\n```bash\nexport RTE_SDK=/home/alireza/packetmill/dpdk/\nexport RTE_TARGET=x86_64-native-linux-gcc\n```\n\n**Make sure to define `RTE_SDK` and `XCHG_SDK` based on the location of `dpdk` and `xchange` directories.**\n\n\n**Note that `NPF` requires all three builds to perform the experiments. It uses `gcc` build (of DPDK v20.02) for packet generation (i.e., default case) and the other two (i.e., X-Change with `clanglto` and DPDK v20.02 with `clang`) for other scenarios at the server side.**\n\nFore more information, please check X-Change [repository][x-change-repo].\n\n### LLVM Toolchain\n\nWe used `LLVM 10.0.0` in our paper. To install it, run the following command:\n\n```bash\nchmod +x llvm-clang.sh\nsudo ./llvm-clang.sh 10\n```\n\nThis command will also create some links to different LLVM tools and clang commands. Check the script (`llvm-clang.sh`) for more details.\n\n### Perf and PMU Tools\n\nWe use `perf` and `pmu-tools` to gather microarchitectural metrics. To install them, run the following commands:\n\n```bash\nsudo apt-get install linux-cloud-tools-$(uname -r) linux-tools-$(uname -r)\ngit clone https://github.com/andikleen/pmu-tools.git\n```\n\n### FastClick\n\nNPF automatically clone and build FastClick for the experiments (based on the testie/npf file), but if you want to compile/build it manually with X-Change repo. You can run the following commands:\n\n```bash\ngit clone --branch packetmill https://github.com/tbarbette/fastclick.git\ncd fastclick\n./configure --disable-linuxmodule --enable-userlevel --enable-user-multithread --enable-etherswitch --disable-dynamic-linking --enable-local --enable-dpdk=$XCHG_SDK --enable-research --disable-task-stats --enable-flow --enable-cpu-load --prefix $(pwd)/build/ --enable-intel-cpu --enable-dpdk-pool --enable-rand-align RTE_TARGET=x86_64-native-linux-clanglto CXX=\"clang++ -flto -fno-access-control\" CC=\"clang -flto\" CXXFLAGS=\"-std=gnu++14 -O3\" LDFLAGS=\"-flto -fuse-ld=lld -Wl,-plugin-opt=save-temps\" RANLIB=\"/bin/true\" LD=\"ld.lld\" READELF=\"llvm-readelf\" AR=\"llvm-ar\" --disable-bound-port-transfer --enable-dpdk-pool --enable-dpdk-xchg --disable-dpdk-packet --disable-dpdk-softqueue\nmake\nsudo make uninstall\nsudo make install\n```\n\n`make install` requires some perl packages. For instance, you might need to run the following command if `make install` fails.\n\n```bash\nsudo cpan File::Which\n```\n\n**Note: if you have already exported X-Change (or DPDK) environment variables, you do not need to pass `RTE_SDK` and/or `RTE_TARGET` in the configure line. However, using `--enable-dpdk=$RTE_SDK` is mandatory if you have a globally installed DPDK (e.g. using apt or ninja install); it forces FastClick to use the right DPDK (e.g., X-Change).**\n\nBuilding FastClick with this configuration uses X-Change by default, i.e., providing `Packet` class to DPDK PMD (`MLX5`). However, it is possible to use other metadata management techniques. The following list summarizes the required compilation flags for different metadata management models. \n\n* **X-Change:** `--enable-dpdk-xchg --disable-dpdk-packet`\n* **Copying:** `--disable-dpdk-packet`\n* **Overlaying:** `--enable-dpdk-packet`\n\nFor more information, please refer to PacketMill's [paper][packetmill-paper]. \n\n\n### Binutils\n\nWe used `bintuils 2.32`. To install/update bintuils, please refer to [here][binutils-page]. **Beware binutils 2.30, which comes with Ubuntu 18.04 has a bug with AVX512. See DPDK commit [e19c6de3][dpdk-commit-link].**\n\n### LLVM Pass - Reordering Pass\n\nWe developed an optimization pass (via LLVM) that reorders the variables/fields of the `Packet` [class][packet-header-fastclick] (i.e., the metadata structure in FastClick) based on the access pattern of the input binary. As our source-code modifications customizes the binary and removes the unused elements (source code), applying our pass together with other optimizations ultimately results in a customized data structure for the input NF configuration.\n\nOur pass parses the whole-program IR code produced by LTO (clang) to finds the references (done by the NF)\nto different variables/fields of the `Packet` class, sorts these variables based on the estimated number of accesses to the variables, and fixes (repairs) the references to the `Packet` class done by the LLVM’s GetElementPtrInst (GEPI) instruction.\n\nAfter applying the pass, the output IR code can be relinked with the dynamic libraries to produce a new binary for FastClick. \n\n**Note that recompiling/relinking the IR code requires removing/striping module flags, we have a simple module pass that performs this task.**\n\n* **Building:** To compile our passes, run the following commands:\n\n```bash\ncd LLVM\nmkdir build\ncd build\ncmake ..\nmake\n```\n\n* **Using:** Compiling FastClick \u0026 X-Change with LTO (clang) while using `plugin-opt=save-temps` flag produces four IR bitcode (i.e., whole-program IR code):\n\n```bash\nclick.0.0.preopt.bc\nclick.0.2.internalize.bc\nclick.0.4.opt.bc\nclick.0.5.precodegen.bc\n```\n\nor \n\n```bash\nembedclick.0.0.preopt.bc\nembedclick.0.2.internalize.bc\nembedclick.0.4.opt.bc\nembedclick.0.5.precodegen.bc\n```\n\nYou can use `llvm-dis` tool to convert them into human-readable LLVM assembly language. For example, try:\n\n```bash\nllvm-dis click.0.5.precodegen.bc -o click.ll\n```\n\nor\n\n```bash\nllvm-dis embedclick.0.5.precodegen.bc -o embedclick.ll\n``` \n\nYou can apply the passes on the `embedclick.ll` via the following commands:\n\n```bash\ncd /home/alireza/fastclick/userlevel/\nopt -S -load /home/alireza/packetmill/LLVM/build/class-stripmoduleflags-pass/libClassStripModuleFlagsPass.so -strip-module-flags click.ll -o click.ll\nopt -S -load /home/alireza/packetmill/LLVM/build/class-handpick-pass/libClassHandpickPass.so -handpick-packet-class click.ll -o click.ll\nopt -S -O3 click.ll -o click.ll\nmake click-opt\n```\n\nor\n\n```bash\ncd /home/alireza/fastclick/userlevel/\nopt -S -load /home/alireza/packetmill/LLVM/build/class-stripmoduleflags-pass/libClassStripModuleFlagsPass.so -strip-module-flags embedclick.ll -o embedclick.ll\nopt -S -load /home/alireza/packetmill/LLVM/build/class-handpick-pass/libClassHandpickPass.so -handpick-packet-class embedclick.ll -o embedclick.ll\nopt -S -O3 embedclick.ll -o embedclick.ll\nmake embedclick-opt\n```\n\n\n### Useful Information about writing LLVM Passes\n\n* Adrian Sampson blog post: http://www.cs.cornell.edu/~asampson/blog/llvm.html\n* Adrian Sampson Skeleton pass: https://github.com/sampsyo/llvm-pass-skeleton\n* Writing an LLVM Pass: https://llvm.org/docs/WritingAnLLVMPass.html\n\n\n## Citing our paper\n\nIf you use PacketMill or X-Change in any context, please cite our [paper][packetmill-paper]:\n\n```bibtex\n@inproceedings{farshin-packetmill,\nauthor = {Farshin, Alireza and Barbette, Tom and Roozbeh, Amir and {Maguire Jr.}, Gerald Q. and Kosti\\'{c}, Dejan},\ntitle = {{PacketMill: Toward per-Core 100-Gbps Networking}},\nyear = {2021},\nisbn = {9781450383172},\npublisher = {Association for Computing Machinery},\naddress = {New York, NY, USA},\nurl = {https://doi.org/10.1145/3445814.3446724},\ndoi = {10.1145/3445814.3446724},\nabstract = {We present PacketMill, a system for optimizing software packet processing, which (i) introduces a new model to efficiently manage packet metadata and (ii) employs code-optimization techniques to better utilize commodity hardware. PacketMill grinds the whole packet processing stack, from the high-level network function configuration file to the low-level userspace network (specifically DPDK) drivers, to mitigate inefficiencies and produce a customized binary for a given network function. Our evaluation results show that PacketMill increases throughput (up to 36.4 Gbps -- 70%) \u0026 reduces latency (up to 101 us -- 28%) and enables nontrivial packet processing (e.g., router) at ~100 Gbps, when new packets arrive \u003e10\\texttimes{} faster than main memory access times, while using only one processing core.},\nbooktitle = {Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems},\npages = {1–17},\nnumpages = {17},\nkeywords = {DPDK, Full-Stack Optimization, FastClick, 100-Gbps Networking, Packet Processing, Commodity Hardware, PacketMill, Compiler Optimizations, LLVM, Metadata Management, Middleboxes, X-Change},\nlocation = {Virtual, USA},\nseries = {ASPLOS 2021}\n}\n```\n\n## Getting Help\n\nIf you have any questions regarding our code or the paper, you can contact [Alireza Farshin][alireza-page] (farshin at kth.se) and/or [Tom Barbette][tom-page] (barbette at kth.se).\n\n\n[x-change-repo]: https://github.com/tbarbette/xchange\n[dpdk-page]: https://www.dpdk.org/\n[packetmill-paper]: https://people.kth.se/~farshin/documents/packetmill-asplos21.pdf\n[packetmill-extended-abstract]: https://people.kth.se/~farshin/documents/packetmill-asplos21-extended_abstract.pdf\n[asplos21-page]: https://asplos-conference.org/2021\n[packetmill-repo]: https://github.com/aliireza/packetmill \n[fastclick-repo]: https://github.com/tbarbette/fastclick\n[tom-page]: https://www.kth.se/profile/barbette\n[alireza-page]: https://www.kth.se/profile/farshin/\n[devirtualize-paper]: https://pdos.csail.mit.edu/~rtm/papers/click-asplos02.pdf\n[ddio-testbed]: https://github.com/aliireza/ddio-bench/blob/master/TESTBED.md \n[npf-setup]: https://github.com/aliireza/ddio-bench/blob/master/TESTBED.md#network-performance-framework-npf-tool\n[npf-readme]: https://github.com/tbarbette/npf/blob/master/README.md\n[binutils-page]: https://www.gnu.org/software/binutils/\n[packet-header-fastclick]: https://github.com/tbarbette/fastclick/blob/master/include/click/packet.hh\n[dpdk-commit-link]: https://github.com/DPDK/dpdk/commit/e19c6de3f2f12d0f49bdf9e787b3f43612c1df3c","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faliireza%2Fpacketmill","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Faliireza%2Fpacketmill","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Faliireza%2Fpacketmill/lists"}