{"id":48661389,"url":"https://github.com/madsys-dev/sosp-paper19-ae","last_synced_at":"2026-04-10T10:11:33.061Z","repository":{"id":186864702,"uuid":"671792010","full_name":"madsys-dev/sosp-paper19-ae","owner":"madsys-dev","description":"The Artifact Evaluation Version of SOSP Paper #19","archived":false,"fork":false,"pushed_at":"2023-08-24T12:40:50.000Z","size":3058,"stargazers_count":1,"open_issues_count":0,"forks_count":0,"subscribers_count":2,"default_branch":"main","last_synced_at":"2023-08-24T14:42:27.148Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/madsys-dev.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null}},"created_at":"2023-07-28T06:33:13.000Z","updated_at":"2023-08-19T06:46:09.000Z","dependencies_parsed_at":"2023-08-08T03:12:12.159Z","dependency_job_id":null,"html_url":"https://github.com/madsys-dev/sosp-paper19-ae","commit_stats":null,"previous_names":["madsys-dev/sosp-paper19-ae"],"tags_count":0,"template":null,"template_full_name":null,"purl":"pkg:github/madsys-dev/sosp-paper19-ae","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madsys-dev%2Fsosp-paper19-ae","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madsys-dev%2Fsosp-paper19-ae/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madsys-dev%2Fsosp-paper19-ae/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madsys-dev%2Fsosp-paper19-ae/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/madsys-dev","download_url":"https://codeload.github.com/madsys-dev/sosp-paper19-ae/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/madsys-dev%2Fsosp-paper19-ae/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31637802,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-10T07:40:12.752Z","status":"ssl_error","status_checked_at":"2026-04-10T07:40:11.664Z","response_time":98,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-10T10:11:29.371Z","updated_at":"2026-04-10T10:11:33.053Z","avatar_url":"https://github.com/madsys-dev.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# CXL-SHM：Non-blocking Partial Failure Resilient Memory Management System\n\nThe efficiency of distributed shared memory (DSM) has been greatly improved by recent hardware technologies. But, the difficulty of distributed memory management can still be a major obstacle to the democratization of DSM, especially when a partial failure of the participating clients (due to crashed processes or machines) should be tolerated.\n\nTherefore,we present **CXL-SHM**, an automatic distributed memory management system based on reference counting. The reference count maintenance in CXL-SHM is implemented with a special era-based non-blocking algorithm. Thus, there are no global blocking, memory leak, double free, and wild pointer problems, even if some participating clients unexpectedly fail without destroying their possessed memory references. We evaluated our system on real CXL hardware with both micro-benchmarks and end-to-end applications, which demonstrate the efficiency of CXL-SHM and the simplicity and flexibility of using CXL-SHM to build efficient distributed applications.\n\nFor more details, please refer to our paper.\n\n\n\n## Requirements and Overview\n\n#### Hardware Environment\n1. Recommended Simulation Platform: General Linux Server (Phyical Machine or Bare Mental Server in Cloud). It would be feasible to utilize a separate NUMA socket's DRAM as a means to emulate the remote CXL hardware. The goal is to facilitate the reproduction of our results. Using remote NUMA to simulate CXL latency is similar to previous works. It's worth noting that our preliminary evaluations indicate a similarity in performance between a remote CXL memory and a cross-NUMA access (Pond [^1] and TBB [^2]). \n2. Original CXL Platform: Intel Linux Server with Sapphire Rapids CPU and FPGA device (Intel Agilex I/Y serial) with R-Tile [^5]. CXL device is configured as devdax mode.\n3. (Optional) RDMA Platform: It can produce the results of baseline in Figure 6. This platform should equip with Mellanox/Nvidia 50GBps ConnectX-5 RDMA NIC.\n\n#### Software Environment\n* Linux Kernel \u003e= 5.10.134\n* OS version \u003e= CentOS 7\n* CMake \u003e= 3.5+\n* Jemalloc: jemalloc = 5.2.1-2.1.al8, jemalloc-devel = 5.2.1-2.1.al8\n* Intel TBB: tbb = 2018.2-9.2.al8, tbb-devel = 2018.2-9.2.al8\n* gcc with C++11 support (10.2.1 in our environment)\n* Main Memory \u003e= 32GB\n* CPU \u003e= 88 cores \\\u0026 \u003e= 44 physical cores (because the maximum thread number is 88 in the evaluations)\n\n#### Dataset\n`LiveJournal` Dataset in SNAP project. It is used in PageRank test and Graph test. You can download it from here [^6].\n\n## Reproduce Steps\n\n#### Prepare Evaluation environment\n\nPlease choose one of the available environments to complete the reproduction process.\n\n##### CXL environment\n* If your environments equip CXL devices with CXL1.1 driver. Please check the device first to ensure the CXL1.1 driver can enumerate the CXL device.\n```shell\nls dax[n].[m]\n```\n*  Next you can use daxctl tool to create a dax (Direct Access) device for evaluations. \n```shell\ndaxctl reconfigure-device -f -m devdax dax[n].[m]\n```\n* Finally, check CXL device is configured as dax device. An example of CXL dax device.\n```shell\ndaxctl list\n```\n\u003cimg src=\".figures/image.png\" width=\"15%\"\u003e\n\n##### Simulation environment\n* General Linux Server must have **at least two NUMA nodes**, because a separate NUMA socket's DRAM is used to emulate the remote CXL hardware. Besides, `numactl` and `numactl-devel` should be installed.\n```shell\nyum install numactl\nyum install numactl-devel\n```\n\n#### Build CXL-SHM\n\nPrerequisites (using root previlege) to install dependency softwares (`jemalloc` and `tbb`).\n```shell\nyum install jemalloc jemalloc-devel\nyum install tbb tbb-devel\n``` \n\nAll following commands assume that you are in the root directory of this repo. \n\nCXL-SHM should know the CPU topologies of your server. So you can run `get-numa-config.sh` script to acquire the details. Before that, you should make sure python3 is installed. After \n```shell\nyum install python3 #(or some other commands)\nsh test/get-numa-config.sh\n```\n\n##### CXL environment\nIf you have the CXL environment (By default, `USE_CXL` option is ON). the cxl device should be configured as dax device. Then you should modify `CXL_DAX_DEV` in `cxlmalloc-internal.h` as the path of dax device (e.g., `/dev/dax[n].[m]`).\n\n```shell\nmkdir build \u0026\u0026 cd build\ncmake ..\nmake\n```\n\n##### Simulation environment\nIf you use emulation environment. As another said, if you don't have the real hardware platform described in CXL environment, we provide emulation methods by disabling the `USE_CXL` option in `CMakeLists.txt`. \n```shell\nmkdir build \u0026\u0026 cd build\ncmake .. -DUSE_CXL=OFF\nmake\n```\n\n#### Dataset\n\nDownload the `LiveJournal` dataset used in PageRank test and Graph test. Make sure you are in the `build` directory.\n```shell\nwget https://snap.stanford.edu/data/soc-LiveJournal1.txt.gz\nyum install gzip\ngunzip soc-LiveJournal1.txt.gz\n```\n\n#### Baseline\n\nRalloc project provides benchmarks to evaluate `Jemalloc`, `Ralloc` and PMDK. Thus we utilize this project as the baseline by adding the support of `Mimalloc`.\n```shell\ngit clone https://github.com/urcs-sync/ralloc.git\ncd ralloc\ngit apply [CXL-SHM home path]/ralloc.patch\n```\n\nPhoenix is a shared-memory implementation of Google's MapReduce model for data-intensive processing tasks.\n```shell\ngit clone https://github.com/kozyraki/phoenix.git\ncd phoenix/sample_apps\ncd kmeans; make\n```\n\nLightning is a high-performance in-memory object store without requiring  IPC overheads on kv operations.\n```\ngit clone https://github.com/danyangz/lightning.git\ndocker build -t lightning .\n```\n\n\n#### Reproduce the data in the Evaluations\n\nTo reproduce the results of this paper, we provide scripts in the ```scripts/``` directory. Also, we have included several screenshots to illustrate the results of **CXL-SHM** in our cxl platform.\n\nTips:\n1. You should use `sh` or `bash` to run all these scripts.\n2. All these scripts should be run in the build directory, thus you can link scripts to the build directory fisrt.\n```shell\nln -s ../script .\n```\n3. To verify that the remote NUMA's DRAM is used as CXL memory, please use `numastat` or other tools. For example, as shown in  this screenshot, high shmem footprint could be seen.\n```shell\nnumastat -m\n``` \n* \u003cimg src=\".figures/image-1.png\" width=\"30%\"\u003e\n\n#### Figure 5\n* To reproduce **CXL-SHM** in Figure 5, use `run_shbench.sh` and `run_threadtest.sh`. Please note that the results are elapsed time, the final results should be transformed according the definition of the original shbench/threadtest (we use the default configurations in Ralloc Project [^3]). \n* shbench: `throughput (MOPS)` = 12.5 / time` and threadtest: `100 / time`.\n```shell\nsh ./script/run_shbench.sh\nsh ./script/run_threadtest.sh\n```\n* To reproduce the comparsions (Ralloc, Jemalloc, Mimalloc) in Figure 5, please use the benchmark in Ralloc Project [^3]. Note that we use the same parameters as Ralloc (1000 iterations, 100000 objects, 8 sz). Before that, if you **use CXL environment** please make sure **CXL device is configured as system ram mode**. Use following commands to do this. After that, use `numactl --membind=[cxl node id]` to ensure that program can allocate CXL memory.\n```shell\ndaxctl reconfigure-device -f -m system-ram dax[n].[m]\n```\n\n* Here is an example of the output after running the command.\n\u003cimg src=\".figures/image-system-ram.png\" width=\"15%\"\u003e\n\n* Here are several steps to reproduce the baselines.\n```shell\ncd test\nmake ALLOC=r\n./run_shbench.sh r\n./run_threadtest.sh r\nmake ALLOC=je\n./run_threadtest.sh je\n./run_shbench.sh je\nmake ALLOC=mi\n./run_shbench.sh mi\n./run_threadtest.sh mi\n```\n\n* CXL environment\n\u003cimg src=\".figures/image-threadtest.png\" width=\"30%\"\u003e\n\u003cimg src=\".figures/image-shbench.png\" width=\"30%\"\u003e\n\n* Simulation environment\n\u003cimg src=\".figures/image-shbench-numa.png\" width=\"30%\"\u003e\n\u003cimg src=\".figures/image-threadtest-numa.png\" width=\"30%\"\u003e\n\n#### Figure 6\n* To reproduce CXL-SHM RPC in Figure 6, use `run-rpc.sh`. Because the performance of CXL-SHM RPC is unrelated to the size of payload, in the different payload size test, we use the data of one client and one server in `run-rpc.sh` as the consistent (invariable) performance.\n```shell\nsh script/run_rpc.sh\n```\n* To reproduce the data of SPSC, please use the larson test (a benchmark for SPSC) in Ralloc Project [^3].\n```shell\n./run_larson.sh je\n```\n\n* We use `ibv_rc_pingpong` tool to mimic RPC RPC. To reproduce the results of RDMA RPC in the left figure, you should run multiple `ibv_rc_pingpong` instance. To reproduce the results of RDMA RPC in the right figure, you should indicate different payload size (2^6~2^22 Bytes) in `ibv_rc_pingpong`. Because the results are presented in latency (e.g., $\\mu s$), you should tranform the data into throughput ($througput = \\frac{1}{latency}$).\n```shell\n# Left Figure\n# In server side\nfor((i=0;i\u003c=[server number];i++)) do ibv_rc_pingpong -n 1000000 -p $((10000+i)) \u003e\u003e rdma-server.txt \u0026; done\n# In client side\nfor((i=0;i\u003c=[server number];i++)) do ibv_rc_pingpong -n 1000000 -p $((10000+i)) \u003e\u003e rdma-client.txt \u0026; done\n\n# Right Figure\n# In server side\nibv_rc_pingpong -s [payloadsize] -n 1000000\n# In client side\nibv_rc_pingpong -s [payloadsize] -n 1000000 [server IP]\n```\n\n\n* CXL environment\n\u003cimg src=\".figures/image-rpc.png\" width=\"30%\"\u003e\n\n* Simulation environment\n\u003cimg src=\".figures/image-rpc-numa.png\" width=\"22%\"\u003e\n\n\n#### Figure 7\n* To reproduce Figure 7, use `run-kmeans.sh` and `run-wc.sh`. \n\u003c!-- * Before that,  in `run-pr.sh`, please setup the `path` as the dataset file (LiveJournal in our evaluations). --\u003e\n```shell\nsh script/run-kmeans.sh\nsh script/run-wc.sh\n```\n\n* To reproduce the comparsions (Phoenix), please use the benchmark in Phoenix Project [^4]. Follow the README to compile the project, and use the apps named ```wordcount``` and ```kmeans``` in the directory ```sample_apps```. Note that ```wordcount``` requires 1GB random generated txt as the input (the same as our implementation of work count). Copy below scripts to the Phoenix directory and run them can reproduce the results. \n```shell\nsh script/run-km-phoenix.sh\nsh script/run-wc-phoenix.sh\n```\n\n* CXL environment\n\u003cimg src=\".figures/image-pr.png\" width=\"30%\"\u003e\n\u003cimg src=\".figures/image-wc.png\" width=\"30%\"\u003e\n\n* Simulation environment\n\u003cimg src=\".figures/image-pr-wc-numa.png\" width=\"30%\"\u003e\n\n#### Figure 8\n* Figure 8a\n  * To reproduce the data of CXL-KV in Figure 8a, use `run_kv.sh`\n  * To reproduce the data of CXL-TBB in Figure 8a, use `run_kv_baseline.sh`\n```shell\nsh script/run_kv.sh\nsh script/run_kv_baseline.sh\n```\n  * To reproduce the data of Lightning, please use docker to construct the environment for compiling and running Lightning (see `README` of Lightning). After that, use `throughput.sh` in `script` dir.\n\n* Figure 8b: To reproduce the data of CXL-KV with different Write/Read Ratio in Figure 8b, use `run_kv_ratio.sh`\n```shell\nsh script/run_kv_ratio.sh\n```\n\n* Figure 8c: To reproduce the data of CXL-KV with different zipf parameter (.5, .9 and .99) in Figure 8c, use `run_kv_zipf.sh`\n```shell\nsh script/run_kv_zipf.sh\n```\n\n* Figure 8d: To reproduce Figure 8d, use `run-smallbank.sh` and `run-tatp.sh`. Besides, we provide two scripts to produce the results of the baselines (`run-smallbank-tbb.sh` and `run-tatp-tbb.sh`).\n```shell\nsh script/run-smallbank.sh\nsh script/run-tatp.sh\nsh script/run-smallbank-tbb.sh\nsh script/run-tatp-tbb.sh\n```\n\n\u003c!-- ![Alt text](image-5.png)\n![Alt text](image-2.png)\n![Alt text](image-3.png)\n![Alt text](image-4.png) --\u003e\n\n* Figure 8e: To reproduce the throughput of Intersect and 2 Hops Query of CXL-Graph in Figure 8e, use `run_graph.sh`. Before that, please setup the `path` as the graph dataset file (LiveJournal in our evaluations). **Note that delete the header lines that contain metadata of the graph dataset.**\n```shell\nsh script/run-graph.sh\n```\n\n* CXL environment\n\u003cimg src=\".figures/image-kv.png\" width=\"30%\"\u003e\n\u003cimg src=\".figures/image-kv-ratio.png\" width=\"30%\"\u003e\n\u003cimg src=\".figures/image-kv-zipf.png\" width=\"30%\"\u003e\n\u003cimg src=\".figures/image-smallbank.png\" width=\"30%\"\u003e\n\u003cimg src=\".figures/image-tatp.png\" width=\"30%\"\u003e\n\u003c!-- \u003cimg src=\".figures/image-graph.png\" width=\"30%\"\u003e --\u003e\n\n* Simulation Environment\n\u003cimg src=\".figures/image-kv-numa.png\" width=\"30%\"\u003e\n\u003cimg src=\".figures/image-kv-ratio-numa.png\" width=\"30%\"\u003e\n\u003cimg src=\".figures/image-sb-tatp-numa.png\" width=\"30%\"\u003e\n\u003c!-- \u003cimg src=\".figures/image-graph-numa.png\" width=\"30%\"\u003e --\u003e\n\n#### Recovery\n\nWe provide fault injection option to test recovery function, do:\n```shell\ncmake .. -DFAULT_INJECTION=ON\nmake\nbash run_recovery_test.sh\n```\nThis is an infinite loop and if the shell does not display a `segmentation fault` alert, it indicates that the recovery process has been successful.\n\n\n## Documantation of CXL-SHM\n\n#### Code structure\n\n| Directory      | Description                                                  |\n| -------------- | :----------------------------------------------------------- |\n| include        | Declaration of System Interface                              |\n| script         | Scaripts which can reproduce the results in the paper        |\n| src            | Implementation details mentioned in this paper               |\n| test           | correctness test, including api test, stress test and fault recovery test |\n| test/benchmark | Benchmarks(threadtest, shbench, rpc micro benchmark, kv, mapreduce(wordcount, pagerank), TATP, SmallBank) |\n\n#### Correctness test\n\nTo run the correctness test including api test and stress test, do:\n\n1. ```ctest```\n\nYou can also run single correctness test, do:\n\n1. ```./cxlmalloc-test-\u003capi|stress\u003e```\n\n#### Benchmark\n\nWe provide several benchmark programs to evaluate the throughput in different scenarios, and there are different parameters for these tests (see details). To run these benchmarks, do:\n\n* Thread test: ``` ./cxlmalloc-benchmark-threadtest [thread number] \u003e threadtest.out```\n* Shbench: ```./cxlmalloc-benchmark-sh6bench [thread number] \u003e shbench.out```\n* RPC:\n  * ```./cxlmalloc-benchmark-rpc_client [thread number] \u003e client.out ```\n  * After ```client.out``` has output (queue is added to share memory), do ```./cxlmalloc-benchmark-rpc_server [thread number] \u003e server.out ```\n* Map Reduce: \n  * Word Count ```./cxlmalloc-benchmark-wc [map number] [reduce number] [data length] \u003e wc.out```\n  * K-Means ```./cxlmalloc-benchmark-km [map number] [reduce number] [points number] [dimensions] [iterations] \u003e km.out```\n* KV: ```./cxlmalloc-benchark-kv [threads number] [read ratio] \u003e kv.out```\n\n* TATP: ``` ./cxlmalloc-benchark-tatp [thread number] \u003e tatp.out```\n* SmallBank: ```./cxlmalloc-benchmark-smallbank [thread number] \u003e smallbank.out```\n* Graph ```./cxlmalloc-benchmark-graph [thread number] [max vertex id] [total edge number] [file path of graph] [Query Mode] \u003e graph.out```\n\n\n[^1]: Li, Huaicheng, et al. Pond: CXL-based memory pooling systems for cloud platforms. Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. 2023.\n[^2]: Maruf, Hasan Al, et al. TPP: Transparent page placement for CXL-enabled tiered-memory. Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 2023.\n[^3]: https://github.com/urcs-sync/ralloc\n[^4]: https://github.com/kozyraki/phoenix.git\n[^5]: https://www.intel.com/content/www/us/en/products/details/fpga/development-kits/agilex/i-series.html\n[^6]: https://snap.stanford.edu/data/soc-LiveJournal1.html\n[^7]: https://github.com/danyangz/lightning\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmadsys-dev%2Fsosp-paper19-ae","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmadsys-dev%2Fsosp-paper19-ae","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmadsys-dev%2Fsosp-paper19-ae/lists"}