{"id":19004544,"url":"https://github.com/osll/ntb-cxl","last_synced_at":"2025-10-29T14:43:34.606Z","repository":{"id":200306201,"uuid":"626329669","full_name":"OSLL/ntb-cxl","owner":"OSLL","description":"NTB/CXL Bridge","archived":false,"fork":false,"pushed_at":"2024-11-12T12:20:53.000Z","size":371,"stargazers_count":4,"open_issues_count":20,"forks_count":0,"subscribers_count":5,"default_branch":"main","last_synced_at":"2025-01-01T19:25:19.203Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"BitBake","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/OSLL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-04-11T08:49:55.000Z","updated_at":"2024-11-12T12:20:59.000Z","dependencies_parsed_at":"2025-01-02T01:45:44.786Z","dependency_job_id":null,"html_url":"https://github.com/OSLL/ntb-cxl","commit_stats":null,"previous_names":["osll/ntb-cxl"],"tags_count":1,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSLL%2Fntb-cxl","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSLL%2Fntb-cxl/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSLL%2Fntb-cxl/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/OSLL%2Fntb-cxl/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/OSLL","download_url":"https://codeload.github.com/OSLL/ntb-cxl/tar.gz/refs/heads/main","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":240027432,"owners_count":19736211,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-08T18:23:36.639Z","updated_at":"2025-10-29T14:43:29.585Z","avatar_url":"https://github.com/OSLL.png","language":"BitBake","funding_links":[],"categories":[],"sub_categories":[],"readme":"# NTB/CXL Bridge for InterVM communication\n\n## Project description\nThis project implements the virtualization of NTB functionality of the IDT 89HPES24NT6AG2 PCI Express Switch device in QEMU. The project has fixed the Linux Kernel version 6.1 and the QEMU version v8.0.2. The goal of the project is to implement a virtual NTB connection between 2 guest machines. The target scenario is the use of RDMA technology through the NTRDMA driver.\n\nImplemented at the moment:\n- VM guest images are based on `core-image-full-cmdline`. Added to the image: `ntb_hw_idt`, `ntb`, `ntb_transport`, `ntb_tool` as a module, `ntb_perf` as a module, `ntrdma` as a module and necessary dependencies.\n- Added `idt-ntb-ivshmem` device to QEMU.\n- Support for NTB registers and functionality:\n  - NTB link functions\n  - Message registers\n  - Doorbell registers\n  - Interrupts and interrupt masking\n  - Memory window translation\n  - Implementation of mw translation in DIR mode\n  - Direct read/write into DMA VM space be i/o operations into mw\n  - Added unsupported by IDT 89HPES24NT6AG2 NTB operations: `mw_set_trans` and `mw_clear_trans` functions (Necessary for `ntb_perf`). So the `ntb_hw_idt` driver was modified.\n- PCIe port 0 and port 2 are configured\n- Only one partition is implemented\n\n\n## Known issues\n- `ntb_perf` test with `use_dma` flag failes (due to the lack of implementation of the DMA controller)\n- `ntrdma` init failes with error (for the same reason as the previous point)\n- if `run_vm` or `run_vms` failes with dynamic link error you need to uncommnet string: `cp $(find ./tmp/work/x86_64-linux/qemu-system-native -type f -name qemu-system-x86_64 | sed -n 2p) $(find ./tmp/work/x86_64-linux/qemu-system-native -type f -name qemu-system-x86_64 | sed -n 1p)` in the `scripts/build_yocto.sh` and rerun qemu build by the command `./run_container.sh --command=build --build=qemu`\n\n\n## Requirements\n- [Docker](https://docs.docker.com/engine/install/)\n\n## Quick start\n1) Build the project in the docker container:\n```\n./run_container.sh --command=enter_qemu_devenv\n./run_container.sh --command=finish_qemu_devenv\n./run_container.sh --command=build\n```\n2) Run VMs:\n```\n./run_container.sh --command=run_vms --qemu-map-ram-to-shm=1000\n```\n3) Connect to the VM1 and VM2 in the another teminals `ssh root@localhost -p7001` and `ssh root@localhost -p7002`\n\n4) Run `ntb_perf` module to test the data transmition on both VMs: `modprobe ntb_perf chunk_order=8 total_order=20`. `chunk_order` is data chunk order [2^n] to transfer, `total_order` is the total data order [2^n] to transfer (a large total order can lead to a MW allocation error due to the BAR size limit and memory limitations of the virtual machine).\n\n5) Check connection info `cat /sys/kernel/debug/ntb_perf/0000\\:00\\:03.0/info`.\n\nExample:\n```\nroot@qemux86-64:~# cat /sys/kernel/debug/ntb_perf/0000\\:00\\:03.0/info \n    Performance measuring tool info:\n\nLocal port 2, Global index 1\nTest status: idle\nPort 0 (0), Global index 0:\n\tLink status: up\n\tOut buffer addr 0x00000000e4157aba\n\tOut buff phys addr 0x00000000fe900000[p]\n\tOut buffer size 0x0000000000100000\n\tOut buffer xlat 0x0000000004b00000[p]\n\tIn buffer addr 0x00000000a1d2914a\n\tIn buffer size 0x0000000000100000\n\tIn buffer xlat 0x0000000004b00000[p]\n```\n\n6) Run test by `echo 0 \u003e /sys/kernel/debug/ntb_perf/0000\\:00\\:03.0/run ` and check the result by `cat /sys/kernel/debug/ntb_perf/0000\\:00\\:03.0/run`.\n\nExample:\n```\nroot@qemux86-64:~# echo 0 \u003e /sys/kernel/debug/ntb_perf/0000\\:00\\:03.0/run \nroot@qemux86-64:~# cat /sys/kernel/debug/ntb_perf/0000\\:00\\:03.0/run \n    Peer 0 test statistics:\n0: copied 1048576 bytes in 1325470 usecs, 0 MBytes/s\n```\n## Usage\n\n### Using docker\n\n`run_container.sh` is a unified starting point for the Docker container.\nThe general syntax is:\n```\n./run_container.sh [OPTIONS]...\n```\nRun `./run_container.sh -h` to see the help.\n\n*NOTE:* `run_container.sh` has one extra option -- `--host-build-dir=DIR`.\nIt represents relative path to the build directory that is mapped to the\ncontainer. This argument is not passed to the container's entrypoint. By\ndefault the `./build_vm_image` directory is used.\n\n### Using locally\n\nFirst of all, install necessary [dependencies](https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html#build-host-packages)\nfor yocto project.\n\nThen use `./scripts/dispatch_docker_command.sh` as entrypoint. This is\nequivalent to using `run_container.sh` script. The rest of the README assumes\nthat the **docker case is used**\n\n## Build project\n\nFirst of all to add the `idt-ntb-ivshmem` device to the qemu it is necessary to create a patch from local source files:\n```\n./run_container.sh --command=enter_qemu_devenv\n./run_container.sh --command=finish_qemu_devenv\n```\n\nAfter `finish_qemu_devenv` command patch will be created and appended to the `meta-ntb-cxl` layer.\n\nRun `./run_container.sh --command=build` to build the project.\n\n## Run VM\n\nTo run VM use command: `./run_container.sh --command=run_vm`\n\nUser credentials to login into the vm:\n- user: ```root```\n- pswd: not set\n\n## Run two connected VMs with `idt-ntb-ivshmem`\n\nRun the following command: `./run_container.sh --command=run_vms --qemu-map-ram-to-shm=1000`. Flag `--qemu-map-ram-to-shm=1000` is necessary for memory window i/o operations to directly read/write into vm memory.\n\nIt uses the `scripts/run_vms.sh` script to run `ivshmem-server` and virtual machines. \n\nQEMU options can be customized via CLI options:\n- `--ivshmem-common-opts`\n- `--cmdline-common`\n- `--common-opts`\n- `--vm1-opts`\n- `--vm2-opts`\n\nThe default behavior is to append whatever is specified in that variables\nto default values.\nTo override the default value instead, suffix the option with `-override`,\nlike `common-opts-override`. Run `--help` for more information\n\nTo see default values of that options, refer to the script itself.\n\nTo connect to the VMs can be used telnet or ssh protocol:\n- telnet port 8001 for VM1 and port 8002 for VM2\n- ssh port 7001 for VM1 and port 7002 for VM2\n\n## QEMU development\n\nQEMU development can be done both on a host system and with Docker\n\nBefore starting, the project should be built and the command\n`./run_container.sh --command=enter_qemu_devenv` should be executed.\nAfter successful execution, the path to checkouted QEMU source repository\nrelative to selected build directory will be printed out\n\nCommand `enter_qemu_devenv` also copies all files from `qemu_src` directory\nto working repository and automatically commits and squashes changes.\n\nMake changes in QEMU and **commit them**. To run bash shell in docker container,\nthe `./run_container --command=shell` command can be used.\n\nThe `./run_container --command=finish_qemu_devenv` command should be used to\nfinish QEMU development. This will format patches from commits and copy them\ninto this repository to `yocto_files`. When QEMU development is finished,\nQEMU rebuild is required. This can be done with\n`./run_container.sh --command=build --build=qemu` command\n\nWhen using host system, you may also try to use various other useful [devtool]\n(https://docs.yoctoproject.org/kernel-dev/common.html#using-devtool-to-patch-the-kernel)\nfunctions.\n\n### QEMU debugging\n\nIt might be useful to be able to read/write to a VM memory directly.\n\nIt's possible using one the following commands:\n```ShellSessinon\n$ ./run_container.sh --command=run_vm --qemu-map-ram-to-shm[=SIZE]\n```\nor\n```ShellSessinon\n$ ./run_container.sh --command=run_vms --qemu-map-ram-to-shm[=SIZE]\n```\n\nThe `=SIZE` is optional. The default is `256`.\nThe value must be provided in mebibytes, without any suffix.\n\nIn case of a single VM, the shared memory will be at `/dev/shm/qemu`.\nIn case of two VMs, the shared memory files will be respectively at\n`/dev/shm/qemu1` and `/dev/shm/qemu2`.\n\nAlso, in case of 2 VMs,\neach VM's allocated memory is equal to a half of the total passed memory size.\n\nIf using docker, to interact with shared memory you must run a shell inside a container:\n```ShellSession\n$ docker exec -it \u003ccontainername\u003e bash\ndocker$ dd if=/dev/shm/qemu bs=1 count=8\ndocker$ ...\n```\n\n## Testing the IDT NTB QEMU device\n\n### `ntb_pingpong`\n\n```ShellSession\n$ ./run_container.sh --command=run_vms\n```\nQEMU should report that something is transferred over out NTB device.\nAlso, the counter in debugfs should increase over time:\n```ShellSession\n$ ssh root@localhost -p7001\n# cat /sys/kernel/debug/ntb_pingpong/*/count\n```\n\n### `ntb_tool`\n\n```ShellSession\n$ ./run_container.sh --command=run_vms --cmdline-common=\"initcall_blacklist=pp_init\"\n```\n(needed to load `ntb_tool` instead of `ntb_pingpong`, both are built-in modules currently)\n\n#### Doorbell register\n\nVM1:\n```ShellSession\n$ ssh root@localhost -p7001\n# cd /sys/kernel/debug/ntb_tool/*\n# echo 's 0xdeadbeef' \u003epeer_db\n```\n\nVM2:\n```ShellSession\n$ ssh root@localhost -p7002\n# cd /sys/kernel/debug/ntb_tool/*\n# cat db\n0xdeadbeef\n```\n\nAlso, setting the peer_mask should work.\n\n#### Message registers\n\nVM1:\n```ShellSession\n# echo 0xdeadbeef \u003epeer0/msg0\n```\n\nVM2:\n```ShellSession\n# cat msg0\n0xdeadbeef\u003c-0\n```\n\nAll message registers (`msg[0-3]`) should work.\n\nSubsequent writes should not be performed if the MSGSTS of the target peer\nwas not cleared (OUTMSGx is set and an interrupt is sent to host which\nattempts to write):\n\n```\nIVSHMEM: Refusing to write to msg register, INMSGSTS0 is non-zero (vm1 0x10000)\n```\n\nINMSGSTSx fields of MSGSTS are set on a write to INMSGx\nand should be unset by the client software on the peer (e.g. `ntb_pingpong`).\nWith `ntb_tool` it can be achieved with the following:\n```ShellSession\n# cat msg_sts\n0x10000\n# echo 'c 0xffffffff' \u003emsg_sts\n# cat msg_sts\n0x0\n```\n\n#### Additional sources\n\nSee [the kernel documentation](\nhttps://docs.kernel.org/driver-api/ntb.html#ntb-tool-test-client-ntb-tool)\nfor the full usage.\nEven more functionality is described in [the source code](\nhttps://elixir.bootlin.com/linux/v6.1.50/source/drivers/ntb/test/ntb_tool.c#L54).\n\n### `ntb_hw_idt` debugfs node\n\nThe driver has its own debugfs node with some useful info:\n```ShellSession\n# cat /sys/kernel/debug/ntb_hw_idt/info\\:0000\\:00\\:03.0\n```\n\nIt can be used to verify that VMs have correct ports.\n\nAlso it's useful for viewing `msg_mask`, which is write-only in `ntb_tool`.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosll%2Fntb-cxl","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fosll%2Fntb-cxl","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fosll%2Fntb-cxl/lists"}