{"id":13508401,"url":"https://github.com/SymbioticLab/Infiniswap","last_synced_at":"2025-03-30T11:31:56.666Z","repository":{"id":50282384,"uuid":"86740036","full_name":"SymbioticLab/Infiniswap","owner":"SymbioticLab","description":"Infiniswap enables unmodified applications to efficiently use disaggregated memory.","archived":false,"fork":false,"pushed_at":"2020-09-26T09:43:05.000Z","size":1195,"stargazers_count":240,"open_issues_count":17,"forks_count":49,"subscribers_count":19,"default_branch":"master","last_synced_at":"2024-11-01T08:33:06.307Z","etag":null,"topics":["memory-disaggregation","rdma"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/SymbioticLab.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2017-03-30T19:21:34.000Z","updated_at":"2024-10-29T14:00:46.000Z","dependencies_parsed_at":"2022-08-25T14:40:33.160Z","dependency_job_id":null,"html_url":"https://github.com/SymbioticLab/Infiniswap","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SymbioticLab%2FInfiniswap","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SymbioticLab%2FInfiniswap/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SymbioticLab%2FInfiniswap/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/SymbioticLab%2FInfiniswap/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/SymbioticLab","download_url":"https://codeload.github.com/SymbioticLab/Infiniswap/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":246314011,"owners_count":20757450,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["memory-disaggregation","rdma"],"created_at":"2024-08-01T02:00:52.545Z","updated_at":"2025-03-30T11:31:51.651Z","avatar_url":"https://github.com/SymbioticLab.png","language":"C","funding_links":[],"categories":["C","others"],"sub_categories":[],"readme":"Infiniswap: Efficient Memory Disaggregation\n====\n\nInfiniswap is a remote memory paging system designed specifically for an RDMA network.\nIt opportunistically harvests and transparently exposes unused memory to unmodified applications by dividing the swap space of each machine into many slabs and distributing them across many machines' remote memory. \nBecause one-sided RDMA operations bypass remote CPUs, Infiniswap leverages the power of many choices to perform decentralized slab placements and evictions.\n\nExtensive benchmarks on workloads from memory-intensive applications ranging  from in-memory databases such as VoltDB and Memcached to popular big data software Apache Spark, PowerGraph, and GraphX show that Infiniswap provides *order-of-magnitude performance improvements* when working sets do not completely fit in memory. \nSimultaneously, it boosts cluster memory utilization by almost 50%. \n\nDetailed design and performance benchmarks are available in our [NSDI'17 paper](https://www.usenix.org/conference/nsdi17/technical-sessions/presentation/gu).\n\nPrerequisites\n-----------\n\nThe following prerequisites are required to use Infiniswap:  \n\n* Software  \n  * Operating system: Ubuntu 14.04 (kernel 3.13.0, also tested on 4.4.0/4.11.0)\n  * Container: LXC (or any other container technologies) with cgroup (memory and swap) enabled  \n  * RDMA NIC driver: [MLNX_OFED 3.2/3.3/3.4/4.1](http://www.mellanox.com/page/products_dyn?product_family=26) (*recommend 4.1*), and select the right version for your operating system. \n\n* Hardware  \n   * Mellanox ConnectX-3/4 (InfiniBand)\n   * An empty and unused disk partition\n\nCode Organization\n-----------\nThe Infiniswap codebase is organized under three directories.\n\n* `infiniswap_bd`: Infiniswap block device (kernel module).\n* `infiniswap_daemon`: Infiniswap daemon (user-level process) that exposes its local memory as remote memory.\n* `setup`: scripts for setup and installation.\n\nImportant Parameters\n-----------\n\nThere are several important parameters to configure in Infiniswap:  \n\n* Infiniswap block device (in `infiniswap_bd/infiniswap.h`)\n  1. `BACKUP_DISK` [disk partition]  \n    It's the name of the backup disk in Infiniswap block device.  \n    How to check the disk partition status and list?  \n      \"sudo fdisk -l\"   \n  2. `STACKBD_SIZE_G` [size in GB]  \n    It defines the size of Infiniswap block device (also backup disk).   \n  3. `MAX_SGL_LEN` [num of pages]  \n    It specifies how many pages can be included in a single swap-out request (IO request).  \n  4. `BIO_PAGE_CAP` [num of pages]  \n    It limits the maximum value of MAX_SGL_LEN.  \n  5. `MAX_MR_SIZE_GB` [size]  \n    It sets the maximum number of slabs from a single Infiniswap daemon. Each slab is 1GB.\n  ```c\n  // example, in \"infiniswap.h\" \n  #define BACKUP_DISK \"/dev/sda4\"  \n  #define STACKBD_SZIE_G 12  // 12GB\n  #define MAX_SGL_LEN 32  // 32 x 4KB = 128KB, it's the max size for a single \"struct bio\" object.\n  #define BIO_PAGE_CAP 32\n  #define MAX_MR_SIZE_GB 32 //this infiniswap block device can get 32 slabs from each infiniswap daemon.\n  ```\n\n* Infiniswap daemon (in `infiniswap_daemon/rdma-common.h`)\n  1. `MAX_FREE_MEM_GB` [size]   \n     It is the maximum size (in GB) of remote memory this daemon can provide (from free memory of the local host).     \n  2. `MAX_MR_SIZE_GB` [size]   \n     It limits the maximum number of slabs this daemon can provide to a single infiniswap block device.   \n     This value should be the same of \"MAX_MR_SIZE_GB\" in \"infiniswap.h\".    \n  3. `MAX_CLIENT` [number]   \n     It defines how many infiniswap block devices a single daemon can connect to.     \n  4. `FREE_MEM_EVICT_THRESHOLD` [size in GB]   \n     This is the \"HeadRoom\" mentioned in our paper.   \n     When the remaining free memory of the host machines is lower than this threshold, infiniswap daemon will start to evict mapped slabs.     \n  ```c\n  // example, in \"rdma-common.h\" \n  #define MAX_CLIENT 32     \n\n  /* Followings should be assigned based on \n  * memory information (DRAM capacity, regular memory usage, ...) \n  * of the host machine of infiniswap daemon.    \n  */\n  #define MAX_FREE_MEM_GB 32    \n  #define MAX_MR_SIZE_GB  32    \n  #define FREE_MEM_EVICT_THRESHOLD 8    \n  ```\n\n#### How to configure those parameters?\n  * If you use the provided installation script (``setup/install.sh``)\n  You can configure those parameters by changing the value of the variables in ``setup/install.sh`` before installation. \n  In ``setup/install.sh``, the definition of the variable and which parameter it maps to have been declared. You can edit its value as needed. For example,\n    ```bash\n    #stackbd (backup) disk size, also the total size of remote memory of this bd\n    #(STACKBD_SIZE), default is 12\n    stackbd_size=12\n    ```\n\n  * If you choose to [build Infiniswap manually](#build), you need to add configuration options to ``configure`` command.\u003cspan id=\"config\"\u003e\u003c/span\u003e\n    You can get the definitions of those options by\n    ```bash\n    # after ./autogen.sh\n    ./configure --help\n    ```\n    See its ``Optional Features``, like:\n    ```\n    --enable-stackbd_size   User defines the size of stackbd (backup) disk which\n                            should be \u003e= the size of remote memory, default is\n                            12\n    ```\n\n    For example, if your Infiniswap block device has 24GB space in both its backup disk and remote memory, you need to\n    ```bash\n    ./configure --enable-stackbd_size=24\n    ``` \n\nHow to Build and Install\n-----------\n\nIn a simple one-to-one experiment, we have two machines (M1 and M2).  \nApplications run in container on M1. \nM1 needs remote memory from M2.  \nWe need to install infiniswap block device on M1, and install infiniswap daemon on M2.  \n\n1. Setup InfiniBand NIC on both machines:  \n```bash  \ncd setup  \n# ./ib_setup.sh \u003cip\u003e    \n# assume all IB NICs are connected in the same LAN (192.168.0.x)\n# M1:192.168.0.11, M2:192.168.0.12\nsudo ./ib_setup.sh 192.168.0.11\n```\n\n2. Compile infiniswap daemon on M2:\n```bash  \ncd setup\n# edit the parameters in install.sh \n./install.sh daemon\n```\n\n3. Install infiniswap block device on M1:  \n```bash  \t\ncd setup\n# edit the parameters in install.sh\n./install.sh bd\n```\n#### Or, how to manually build Infiniswap? \u003cspan id=\"build\"\u003e\u003c/span\u003e\n  * Infiniswap daemon  \n  ```bash  \t\n  cd infiniswap_daemon\n  ./autogen.sh\n  ./configure [options] \n  make\n  ``` \n\n  * Infiniswap block device\n  ```bash  \t\n  cd infiniswap_bd\n  ./autogen.sh\n  ./configure [options] \n  make\n  sudo make install\n  ``` \n\n  If you want to change the parameters of Infiniswap, you can add options when executing ``configure``. \n  Please read [how to add configure options](#config) for details.\n\nHow to Run\n-----------\n1. Start infiniswap daemon on M2:  \n    ```bash  \t\n    cd infiniswap_daemon   \n    # ./infiniswap-daemon \u003cip\u003e \u003cport\u003e \n    # pick up an unused port number\n    ./infiniswap-daemon 192.168.0.12 9400\n    ```\n\n2. Prepare server (portal) list on M1:  \n    ```  \n    # Edit the port.list file (\u003cinfiniswap path\u003e/setup/portal.list)\n    # portal.list format, the port number of each server is assigned above.  \n    Line1: number of servers\n    Line2: \u003cserver1 ip\u003e:\u003cport\u003e  \n    Line3: \u003cserver2 ip\u003e:\u003cport\u003e\n    Line4: ...\n    ```\n    ```bash  \n    # in this example, M1 only has one server\n    1\n    192.168.0.12:9400\n    ```\n\n3. Disable existing swap partitions on M1:\n    ```bash  \t\n    # check existing swap partitions\n    sudo swapon -s\n\n    # disable existing swap partitions\n    sudo swapoff \u003cswap partitions\u003e\n    ```\n\n4. Create an infiniswap block device on M1:  \n    ```bash  \t\n    cd setup\n    # create block device: nbdx-infiniswap0\n    # make nbdx-infiniswap0 a swap partition\n    sudo ./infiniswap_bd_setup.sh\n    ```\n\n    ```bash  \t\n    # If you have the error: \n    #   \"insmod: ERROR: could not insert module infiniswap.ko: Invalid parameters\"\n    # or get the following message from kernel (dmesg):\n    #   \"infiniswap: disagrees about version of symbol: xxxx\"\n    # You need a proper Module.symvers file for the MLNX_OFED driver (kernel module)\n    #\n    cd infiniswap_bd\n    make clean\n    cd ../setup\n    # Solution 1 (copy the Module.symvers file from MLNX_OFED dkms folder):\n    # provide mlnx_ofed_version: 3.2,3.3,3.4,4.1, or not (default is 4.*)\n    ./get_module.symvers.sh {mlnx_ofed_version}\n    # ./get_module.symvers.sh 4.1\n    # Or solution 2 (generate a new Module.symvers file)\n    ./create_Module.symvers.sh {mlnx_ofed_version}\n    # Then, recompile infiniswap block device from step 3 in \"How to Build and Install\"\n    ```\n\n5. Configure memory limitation of container (LXC)  \n    ```bash  \t\n    # edit \"memory.limit_in_bytes\" in \"config\" file of container (LXC)\n\n    # For example, this container on M1 can use 5GB local memory at most.\n    # Additional memory data will be stored in the remote memory provided by M2.   \n    lxc.cgroup.memory.limit_in_bytes = 5G\n    ```\n\nNow, you can start your applications (in container).     \nThe extra memory data from applications will be stored in remote memory.   \n\nFAQ\n----------\n1. Does infiniswap support transparent huge page?   \n**Yes.**\nInfiniswap relies on the swap mechanism in the original Linux kernel.\nCurrent kernel (we have tested up to 4.10) splits the huge page into basic pages (4KB) before swapping out the huge page.  \n(In `mm/vmscan.c`, `shrink_page_list()` calls `split_huge_page_to_list()` to split the huge page.)   \nTherefore, whether transparent huge page is enabled or not makes no difference for infiniswap.   \n\n2. Can we use Docker container, other than LXC?    \n**Yes.**\nInfiniswap requires container-based environment. \nHowever, it has no dependency on LXC. Any container technologies that can limit memory resource and enable swapping should be feasible.  \nWe haven't tried Docker yet. If you find any problems when running infiniswap in a Docker environment, please contact us.  \n\n3. Invalid parameters error when insert module?\nThere are two ways of compiling infiniswap; using 1) inbox driver 2) Mellanox OFED. \nWhen you use inbox driver, you can compile/link against kernel headers/modules.\nWhen you use Mellanox OFED, you need to compile/link against OFED headers/modules.\nThis should be handled by configure file, and refer the Makefile that links OFED modules.\n\n4. Others issues about compatibility\n    * `lookup_bdev()` has different input arguments in the [kernel patch](https://www.redhat.com/archives/dm-devel/2016-April/msg00372.html).\n      By default, we assume the patch is **not installed**. If you OS has this patch, you should:\n        * If you use ``setup/install.sh``, please set \n          ```bash\n          # setup/install.sh\n          have_lookup_bdev_patch=1  #the default value is 0.\n          ```\n        * **Or,  if you build ``infiniswap_bd`` manually**, add ``--enable-lookup_bdev`` in the configuration step.\n\n\n    \n\nContact\n-----------\nThis work is by [Juncheng Gu](http://web.eecs.umich.edu/~jcgu/), Youngmoon Lee, Yiwen Zhang, [Mosharaf Chowdhury](http://www.mosharaf.com/), and [Kang G. Shin](https://web.eecs.umich.edu/~kgshin/). \nYou can email us at `infiniswap at umich dot edu`, file issues, or submit pull requests.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSymbioticLab%2FInfiniswap","html_url":"https://awesome.ecosyste.ms/projects/github.com%2FSymbioticLab%2FInfiniswap","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2FSymbioticLab%2FInfiniswap/lists"}