{"id":31815824,"url":"https://github.com/xilinx/open-nic-shell","last_synced_at":"2026-02-16T02:41:23.253Z","repository":{"id":39986820,"uuid":"343549926","full_name":"Xilinx/open-nic-shell","owner":"Xilinx","description":"AMD OpenNIC Shell includes the HDL source files","archived":false,"fork":false,"pushed_at":"2025-01-02T19:58:15.000Z","size":17695,"stargazers_count":104,"open_issues_count":18,"forks_count":68,"subscribers_count":9,"default_branch":"main","last_synced_at":"2025-01-02T20:32:47.984Z","etag":null,"topics":["datacenter","fpga","network-acceleration","smartnic","verilog"],"latest_commit_sha":null,"homepage":"","language":"SystemVerilog","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/Xilinx.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG.md","contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE.txt","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2021-03-01T20:37:05.000Z","updated_at":"2025-01-02T19:58:19.000Z","dependencies_parsed_at":"2025-01-02T20:28:21.857Z","dependency_job_id":"d96aa8ac-e972-4874-bf86-86436d070e57","html_url":"https://github.com/Xilinx/open-nic-shell","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/Xilinx/open-nic-shell","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xilinx%2Fopen-nic-shell","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xilinx%2Fopen-nic-shell/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xilinx%2Fopen-nic-shell/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xilinx%2Fopen-nic-shell/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/Xilinx","download_url":"https://codeload.github.com/Xilinx/open-nic-shell/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/Xilinx%2Fopen-nic-shell/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":279006755,"owners_count":26084178,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","status":"online","status_checked_at":"2025-10-11T02:00:06.511Z","response_time":55,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["datacenter","fpga","network-acceleration","smartnic","verilog"],"created_at":"2025-10-11T09:25:12.141Z","updated_at":"2026-02-16T02:41:23.245Z","avatar_url":"https://github.com/Xilinx.png","language":"SystemVerilog","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AMD OpenNIC Shell\n\nThis is one of the three components of the\n[OpenNIC project](https://github.com/Xilinx/open-nic.git).  The other components are: \n- [OpenNIC driver](https://github.com/Xilinx/open-nic-driver.git) and\n- [OpenNIC DPDK](https://github.com/Xilinx/open-nic-dpdk.git).\n\nOpenNIC shell delivers an FPGA-based NIC shell with 100Gbps Ethernet ports.  The\nlatest version is built with Vivado 2020.x, 2021.x or 2022.1.  Currently, the supported boards include:\n\n- Xilinx Alveo U50, and\n- Xilinx Alveo U55N, and\n- Xilinx Alveo U55C, and\n- Xilinx Alveo U200, and\n- Xilinx Alveo U250, and\n- Xilinx Alveo U280, and\n- Xilinx Alveo U45N\n\n| Notes: |\n|:---|\n|* In the Alveo U50 version only, Vivado may issue critical warnings regarding the power margin for the MGTYAVtt with respect to a margin on the 4A rail limit. While the U50's open-nic-shell MGT current (~3.97A) is still slightly below the rail's limit, this will go outside of the 10% margin described in the U50 board files.  The U50 version has worked in a lab setting but with minimal testing.  While this issue is considered to have low risk, please be aware of this condition and proceed if that risk is acceptable.\n|* Vivado 2022.2 uses QDMA v5.0. Driver may need to be upgraded to support QDMA 5.0.\n|* Also, starting from OpenNIC 1.0, the support for Bittware SoC-250 is obsolete and no longer maintained.\n\n\nThe NIC shell consists of skeleton components which implement host and Ethernet\ninterfaces and two user logic boxes that wraps up user RTL plugins.  Its\narchitecture is shown in the figure below.\n\n    -----  -----------------------------------------------\n    |   |  |            System Configuration             | \n    |   |  -----------------------------------------------\n    |   |     |         |         |         |         |  AXI-lite 125MHz\n    |   |     V         V         V         V         V\n    |   |  -------   -------   -------   -------   -------\n    |   |  |     |   |     |   |     |   |     |   |     |\n    | P |  |  Q  |==\u003e| Box |==\u003e|  A  |==\u003e| Box |==\u003e|  C  |\n    | C |  |  D  |   |  @  |   |  D  |   |  @  |   |  M  |\n    | I |  |  M  |   | 250 |   |  A  |   | 322 |   |  A  |\n    | E |  |  A  |\u003c==| MHz |\u003c==|  P  |\u003c==| MHz |\u003c==|  C  |\n    |   |  |     | | |     | | |     | | |     | | |     |\n    -----  ------- | ------- | ------- | ------- | -------\n                   |         |         |         |\n                   -----------         -----------\n                 AXI4-stream 250MHz  AXI4-stream 322MHz\n\nThe shell skeleton has the following components.\n\n- QDMA subsystem.  It includes the Xilinx QDMA IP and RTL logic that bridges the\n  QDMA IP interface and the 250MHz user logic box.  The interfaces between QDMA\n  subsystem and the 250MHz box use a variant of the AXI4-stream protocol.  Let\n  us refer the variant as the 250MHz AXI4-stream.\n  \u003e - U45N has two QDMA subsystems. One for the host CPU, another \n  \tfor onboard Arm CPU. Two sets of AXI4-stream interfaces between QDMA\n\tsubsystems and the 250MHz box.\n- CMAC subsystem.  It includes the Xilinx CMAC IP and some wrapper logic.\n  OpenNIC shell supports either 1 or 2 CMAC ports.  In the case of 2 CMAC ports,\n  there are two instances of CMAC subsystems with dedicated data and control\n  interfaces.  The CMAC subsystem runs at 322MHz and connects to the 322MHz user\n  logic box using a variant of the AXI4-stream protocol.  Similarly, let us\n  refer the variant as the 322MHz AXI4-stream.\n- Packet adapter.  It is used to convert between the 250MHz AXI4-stream and\n  322MHz AXI4-stream.  On both TX and RX paths, the packet adapter serves as a\n  packet-mode FIFO, which buffers the whole packet before sending out.  On the\n  RX path, it also restores the back-pressure capability which is missing from\n  the CMAC subsystem interface.\n- System configuration.  It implements a reset mechanism and allocates the\n  register addresses for each components.  The register interface uses AXI4-lite\n  protocol and runs at 125MHz, which is phase-aligned with the 25MHz clock.\n\nThere are 2 user logic boxes, one running at 250MHz and the other at 322MHz.\nEach has a AXI-lite interface for register, 2 pairs of slave and master\nAXI4-stream interfaces for TX and RX respectively.  User RTL plugins are\nresponsible for implementing the handling of these interfaces.\n\n## Repo Structure\n\nThe `open-nic-shell` repository is organized as follows.\n\n    |-- open-nic-shell --\n        |-- constr --\n            |-- au50 --\n\t        |-- au55c --\n\t        |-- au55n --\n            |-- au200 --\n            |-- au250 --\n            |-- au280 --\n\t        |-- U45N --\n            |-- ... --\n        |-- plugin --\n            |-- p2p --\n        |-- script --\n            |-- board_settings --\n            |-- build.tcl\n            |-- ...\n        |-- src --\n            |-- box_250mhz --\n            |-- box_322mhz --\n            |-- cmac_subsystem --\n            |-- packet_adapter --\n            |-- qdma_subsystem --\n            |-- system_config --\n            |-- utility --\n            |-- open_nic_shell.sv\n            |-- ...\n        |-- LICENSE.txt\n        |-- README.md\n        |-- ...\n\nMost of the directories are self-explanatory.  The code under `src` contains the\nskeleton components and the \"empty\" boxes.  Sample plugins are available under\nthe `plugin` directory.\n\n## How to Build\n\nOpenNIC shell is built by running the Tcl script `build.tcl` under `script` in\nVivado.  Depending on the target device, the build script generates proper files\nfor flash programming.\n\nIt is recommended to build the design with Internet connection, as it relies on\nupdated Xilinx board files, accessible through [Xilinx Board\nStore](https://github.com/Xilinx/XilinxBoardStore).  The build script\nautomatically updates Vivado against this repository.  See the below section for\nbuild without Internet/Github access.\n\n### Build Script Options\n\nTo start building the shell, run the following command under `script` with a\nproper `MODE` choice (i.e., `tcl`, `batch` or `gui`).\n\n    vivado -mode MODE -source build.tcl -tclargs [-OPTION VALUE] ...\n\nA list of options are available to configure the build process and customize\nthe design parameters.\n\n    # Build options\n\n    -board_repo  PATH\n                 path to local Xilinx board store repository for offline build.\n                 This option is used when Vivado is unable to connect to github\n                 and update the board repository.\n\n    -board       BOARD_NAME\n                 supported boards include:\n                 - au250, and\n                 - au280, and\n                 - au200, and\n                 - au55c, and\n                 - au55n, and\n                 - au50,  and\n\t\t\t\t - U45N.\n\n    -tag         DESIGN_TAG\n                 string to identify the build.  The tag, along with the board\n                 name, becomes part of the build directory name.\n\n    -overwrite   0 (default), 1\n                 indicate if the script should overwrite existing build results.\n\n    -jobs        [1, 32] (default to 8)\n                 number of jobs for synthesis and implementation.\n\n    -synth_ip    0, 1 (default)\n                 indicate if IPs are out-of-box synthesized after creation.\n\n    -impl        0 (default), 1\n                 indicate if the script runs towards bitstream generation.  If\n                 set to 0, the script only creates the project and do not launch\n                 any run.\n\n    -post_impl   0 (default), 1\n                 indicate if MCS file is generated after bitstream generations.\n\n    -user_plugin PATH\n                 path to the user plugin repository.\n\n    # Design parameters\n\n    -build_timestamp VALUE\n                     VALUE should be an 8-digit hexdecimal value without prefix.\n                     It serves as the timestamp to identify the build and is\n                     written into the shell register 0x0.  If not specified, the\n                     date and time of the build is recorded using the format\n                     MMDDhhmm, where MM is for month, DD for day, hh for hour\n                     and mm for minute.\n\n    -min_pkt_len     [64, 256] (default to 64)\n                     minimum packet length.\n\n    -max_pkt_len     [256, 9600] (default to 1514)\n                     maximum packet length.\n\n    -use_phys_func   0, 1 (default)\n                     indicates if the QDMA H2C and C2H AXI4-stream interfaces are\n                     included in the 250MHz user logic box.  A common scenario\n                     for not using them is networking accelerators without DMA.\n                     Regardless the value of this option, the QDMA IP is always\n                     present in the shell since it also provide the AXI-lite\n                     interfaces for register access.\n\n    -num_phys_func   [1, 4] (default to 1)\n                     number of QDMA physical functions per QDMA subsystem.\n\n    -num_qdma        1 (default), 2\n                     number of QDMA subsystems, subjects to the board model.\n\n    -num_queue       [1, 2048] (default to 512)\n                     number of QDMA queues.\n\n    -num_cmac_port   1 (default), 2\n                     number of CMAC ports, subjects to the board model.\n\n### Build Process\n\nThe build process involves four steps.\n\n1. IP creation and optionally, out-of-box synthesis.\n2. Design project setup.\n3. Synthesis, implementation and bitstream generation.\n4. Post-processing.\n\nBy default, the script completes the first two steps, producing a Vivado project\nunder the build directory.  This can be customized using the `-synth_ip`,\n`-impl` and `-post_impl` options.\n\n- If `-synth_ip` is set to 0, the IP out-of-box synthesis is deferred.\n- If `-impl` is set to 1, the third step is performed.\n- If `-post_impl` is set 1, the post-processing step is performed after\n  bitstream generation.\n\nThe build directory is located under `build` and named as `[BOARD]_[TAG]`.\nUnder the build directory, there is a text file, `DESIGN_PARAMETERS`, which\ncontains the parameters passed to the RTL top-level.  All the IP files are\nstored under `vivado_ip`.  The shell project is under `open_nic_shell`.\n\nThe following Verilog macros are defined and made available to the RTL source\ncode.\n\n- The `__synthesis__` macro.\n- Board name, either `__au250__`, `__au280__`, `__au50__`, `__au55c__`, `__au55n__` or `__au45n__`.\n\n### Build without Github Access from Vivado\n\nIf Vivado does not have access to Github, we need to have a local copy of the\n[Xilinx Board Store](https://github.com/Xilinx/XilinxBoardStore) repository, and\npass the path to the build script via the `-board_repo` option.\n\n## User Plugin Integration\n\nOpenNIC shell provides 2 user logic boxes for instantiating custom RTL logic,\none running at 250MHz and the other at 322MHz.  To build with custom plugins,\npass the path of the plugin repository to the `-user_plugin` argument of the\nbuild script.  Default plugins are available for both boxes under\n`plugin/p2p/box_250mhz` and `plugin/p2p/box_322mhz` respectively, which does\nsimple port-to-port connection.  If users are interested in only one box, they\nshould instantiate the default plugin in the other one.\n\nEach box requires a top-level wrapper for all the user plugins instantiated in\nthat box.  In other words, only one module should be instantiated in each box.\nThere are a few rules on how to structure the `-user_plugin`.  At a minimum\nlevel, it should look like the following.\n\n    |-- USER_PLUGIN_DIR --\n        |-- box_250mhz --\n            |-- user_plugin_250mhz_inst.vh\n            |-- box_250mhz_address_map_inst.vh\n            |-- box_250mhz_address_map.v\n            |-- box_250mhz_axi_crossbar.tcl\n        |-- box_322mhz --\n            |-- user_plugin_322mhz_inst.vh\n            |-- box_322mhz_address_map_inst.vh\n            |-- box_322mhz_address_map.v\n            |-- box_322mhz_axi_crossbar.tcl\n        |-- ... --\n        |-- build_box_250mhz.tcl\n        |-- build_box_322mhz.tcl\n        |-- ...\n\nThe files under `box_250mhz` and `box_322mhz` are glue code that connects the\nuser plugins to OpenNIC shell.\n\n- `user_plugin_XXXmhz_inst.vh` is a Verilog header file that instantiates the\n  top-level wrapper.  It is included into `src/box_XXXmhz/box_XXXmhz.sv`.\n- `box_XXXmhz_address_map_inst.vh` is a Verilog header file that instantiates\n  the `box_XXXmhz_address_map` module.  It is included into\n  `src/box_XXXmhz/box_XXXmhz.sv`.\n- `box_XXXmhz_address_map.v` implements the register address mapping in\n  `box_XXXmhz`.\n- `box_XXXmhz_axi_crossbar.tcl` creates the AXI crossbar IP instantiated in\n  `box_XXXmhz_address_map.v`.\n\nCurrently, these files need to be created and modified manually.  In the next\nrelease, they will be auto-generated through a configuration file.\n\nFor each box, the build script performs the following steps.\n\n1. Source `box_XXXmhz_axi_crossbar.tcl`.\n2. Read `box_XXXmhz_address_map.v`.\n3. Add `USER_PLUGIN_DIR/box_XXXmhz` to the `include_dirs` property.\n4. Source `build_box_XXXmhz.tcl`.\n\nTo use the default plugin (i.e., `plugin/p2p`) in one of the boxes, remove the\ncorresponding `box_XXXmhz` directory and `build_box_XXXmhz.tcl`.\n\nWhen the design is configured with two QDMA subsystems, two sets of AXI4-stream \ninterfaces are provided to box_250MHz. The default p2p plugin of box_250MHz has \none ingress switch and one egress switch per QDMA physical function. For example, \nP2P plugin of U45N has totaly four AXI4-stream switches. To select the data \npath between MAC and two QDMA subsystems, AXI4-stream switch control registers\nare used. The user can select one QDMA subsystem to be used exclusively for \ntransmitting or receiving traffic. Please refer to [AXI4-Stream Switch Control \nRegister](https://docs.xilinx.com/r/en-US/pg085-axi4stream-infrastructure/Control-Register) \nof PG085 for more details.\n\nFor example:\n\nData path: MAC \u003c-\u003e QDMA subsystem 0 (On U45N, it is connected to host CPU)\nSelect QDMA subsystem 0:\n```bash\n# BDF is the bus ID of first physical function of QDMA's\nbdf=\u003cPCIe BDF\u003e #e.g. d9\n# Ingress port 0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100040 w*1 0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100000 w*1 0x2\n# Egress port 0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x1000c0 w*1 0x0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100080 w*1 0x2\n# Ingress port 1\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100140 w*1 0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100100 w*1 0x2\n# Egress port 0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x1001c0 w*1 0x0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100180 w*1 0x2\n```\n\nData path: MAC \u003c-\u003e QDMA subsystem 1 (On U45N, it is connected to onboard Arm CPU)\nTo select QDMA subsystem 1:\n```bash\n# BDF is the bus ID of first physical function of QDMA's\nbdf=\u003cPCIe BDF\u003e #e.g. d9\n# Ingress port 0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100040 w*1 0x80000000\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100044 w*1 0x0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100000 w*1 0x2\n# Egress port 0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x1000c0 w*1 0x1\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100080 w*1 0x2\n# Ingress port 1\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100140 w*1 0x80000000\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100144 w*1 0x0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100100 w*1 0x2\n# Egress port 0\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x1001c0 w*1 0x1\npcimem /sys/bus/pci/devices/0000\\:${bdf}\\:00.0/resource2 0x100180 w*1 0x2\n```\n\n## Shell Interface\n\nShell and user logic boxes communicates through 3 types of interfaces.\n\n- AXI-lite interface running at 125MHz for register access.\n- AXI4-stream interface running at either 250MHz or 322MHz for data path.\n- Synchronous reset interface running at 125MHz.\n\nThe 125MHz and 250MHz clock domains are phase aligned.  Signals can be sampled\nacross each domain without clock domain crossing.  On the other hand, do note\nthat different clock frequencies can lead to double sampling or missing samples.\n\nThe AXI-lite interface follows the standard AXI4-lite protocol without `wstrb`,\n`awprot` and `arprot`.  At the system level, the address ranges for the 2 boxes\nare\n\n- 0x100000 - 0x1FFFFF for the 322MHz box, and\n- 0x200000 - 0x2FFFFF for the 250MHz box.\n\nThe 250MHz and 322MHz AXI4-stream interfaces have slightly different semantics.\nThe 250MHz interface has the following signals.\n\n- `tvalid`, 1 bit: same as standard AXI4-stream protocol.\n- `tdata`, 512 bits: data maps from lower to upper bytes.\n- `tkeep`, 64 bits: null bytes are only allowed when both `tvalid` and `tlast`\n  are asserted and cannot be followed by other data bytes for the same packet.\n  For example, a 96B packet has a `tkeep` value of 0xFFFFFFFFFFFFFFFF in the\n  first beat, and 0x00000000FFFFFFFF in the second beat.\n- `tlast`, 1 bit: same as standard AXI4-stream protocol.\n- `tuser_size`, 16 bits: field to specify packet size.  It contains the number\n  of bytes in the packet and must remain valid and unchanged if `tvalid` is\n  asserted.\n- `tuser_src`, 16 bits: source of the packet.\n- `tuser_dst`, 16 bits: destination of the packet.\n- `tuser_user`, N bits: side-band user data.\n- `tready`, 1 bit: same as standard AXI4-stream protocol.\n\nFor the `tkeep` signal, the interface assumes its presence depending on the\ndirection of a packet.\n\n- For packets exiting the shell, it is guaranteed that `tkeep` is consistent\n  with `tuser_size`.  This means that `tkeep` consists of all 1s when `tvalid`\n  is asserted and `tlast` is de-asserted and shows a bit-mask for the valid\n  bytes in the beat when both `tvalid` and `tlast` are asserted.\n- For packets entering the shell, `tkeep` can be optionally set to all 1s\n  regardless of the value of `tuser_size`.  This allows user plugins to drop\n  `tkeep` in their implementation.  If `tkeep` is not set to all 1s, it must be\n  consistent with `tuser_size`.\n  \nFor `tuser_src` and `tuser_dst`, they have the following format.  For packets\nexiting the shell, `tuser_src` is marked accordingly and `tuser_dst` is all 0s.\n\n    ----------------------------------------------------\n    | 15                         6 |    4 |          0 |\n    |------------------------------|------|------------|\n    | P  P  P  P  P  P  P  P  P  P | R  R | F  F  F  F |\n    |------------------------------|------|------------|\n    |            MAC ports         | Rsvd |  PCIe PFs  |\n    ----------------------------------------------------\n\nThe 322MHz interface is more restrictive due to requirements from CMAC IP.  It\nhas the following signals.\n\n- `tvalid`, 1 bit: no de-assertion in the middle of a packet.  In other words,\n  once `tvalid` is asserted to indicate the start of packet, it must remain\n  asserted until `tlast`.\n- `tdata`, 512 bits: same as in the 250MHz interface.\n- `tkeep`, 64 bits: same as in the 250MHz interface.\n- `tlast`, 1 bit: same as in the 250MHz interface.\n- `tuser_err`, 1 bit: indicates if the packet contains an error.\n- `tready`, 1 bit: same as in the 250MHz interface.  But packets entering the\n  shell, i.e., from the RX side of CMAC IPs, the `tready` signal is not present\n  and the master side assumes that `tready` is always asserted.\n\n## Behavioral simulation\n\nCurrently, cocotb (https://www.cocotb.org/) and modelsim (https://eda.sw.siemens.com/en-US/) are used for simulation. cocotb allows writing testbenches in python.\n\n- Quick tutorial on modelsim: https://users.ece.cmu.edu/~jhoe/doku/doku.php?id=a_short_intro_to_modelsim_verilog_simulator\n- Reference testbenches using cocotb: https://github.com/alexforencich/verilog-axis/blob/master/tb/axis_switch/test_axis_switch.py\n\n### Building for simulation\nExample command:\n```\nvivado -mode tcl -source ./build.tcl -tclargs \\\n  -board au280 \\\n  -num_cmac_port 2 -num_phys_func 2 \\\n  -sim 1 \\\n  -sim_lib_path $HOME/opt/xilinx_sim_libs/Vivado2021.2/compile_simlib \\\n  -sim_exec_path $HOME/opt/modelsim/modelsim-se_2020.1/modeltech/linux_x86_64 \\\n  -sim_top p2p_250mhz\n```\n- `sim 1` builds simulation sources.\n- `sim_lib_path` specifies libraries used by Xilinx IPs.\n- If the directory pointed by `sim_lib_path` is empty, the build script will compile and install the sim libraries. This step is one time and can take ~hours. This simulation libraries are global and can be shared across projects.\n - `sim_exec_path` is path to modelsim installation (i.e., the directory containing the vsim executable).\n\nThis command will create simulation sources in `build/\u003cboard\u003e_\u003ctag\u003e/open_nic_shell/open_nic_shell.sim/sim_1/behav/modelsim`. These sources can be used to test any module instantiated within `sim_top`.\n\n### Running simulation\n\n```\ncd build/\u003cboard\u003e_\u003ctag\u003e/open_nic_shell/open_nic_shell.sim/sim_1/behav/modelsim\n\n# Symlink helper scripts\nln -s \u003cpath-to\u003e/open-nic-shell/script/tb/* ./\n# Example:\nln -s ../../../../../../../script/tb/* ./\n\n# Symlink the test bench files\nln -s \u003cpath-to\u003e/open-nic-shell/plugin/p2p/box_250mhz/tb/\u003cmodule-to-test\u003e/* ./\n# Example:\nln -s ../../../../../../../plugin/p2p/box_250mhz/tb/* ./\n\n# Compile the simulation sources using modelsim\n./compile.sh  # (This script is autogenerated during building step by Vivado)\n\n# Run the simulation\nDUT=\u003cmodule-to-test\u003e GUI=0 DEBUG=0 ./run.sh  # run.sh is linked from the script/tb directory\n# Example\nDUT=p2p_250mhz_wrapper ./run.sh\n## Set GUI=1 to see waveform\n```\nNote, in the above example, the top level module specified to Vivado was `p2p_250mhz`, while the top module being simulated is `p2p_250mhz_wrapper`. This can be done since we also copied `p2p_250mhz_wrapper.sv`. In general any module that is included in the top module specified to Vivado can be simulated with the exported sources.\n\n### How the simulation works\n#### Simulation setup\nOpen nic shell uses Xilinx IPs, these cannot be directly simulated by third party simulators. Third party simulators require Vivado to compile simulation libraries.\n\nFor any project, Vivado can output simulation sources for a set of supported simulators. Vivado will understand the dependencies in the project and created sources for simulating a specified top level module.\n\nThese sources are then compiled by the simulator (modelsim in this case). The run file executes modelsim and also tells it to use cocotb (using the pli flag), cocotb knows the test bench file using the exported env variables in run.sh file. (See https://docs.cocotb.org/en/stable/custom_flows.html for reference on how to tell modelsim that cocotb is being used).\n\n#### Simulation execution\nThe python test bench executes python code until it reaches a point where it needs simulation of the verilog design, at that point modelsim takes over until some signals need to be set by the python test bench and this process continues.\n\nNote: There is no fundamental restriction to use above tools. One can write a testbench in verilog to simulate parts of open nic shell using Vivado simulator (https://docs.xilinx.com/r/en-US/ug900-vivado-logic-simulation).\n\n## Programming FPGA\n\nAfter bitstream generation, FPGA can be programming in two ways.\n\n1. Program the device directly.  The FPGA configuration will lose after reboot.\n2. Program and boot from the configuration memory.\n\nBecause the shell bitstream contains a PCI-e IP core, both approaches will cause\nthe lost of PCI-e link.  It could be fatal depending on the model of host\nmotherboards.  For example, for Dell servers, losing discovered PCI-e links\ncould lead to a forced reboot triggered by iDRAC.\n\nTo avoid this, use the Bash script `script/setup_device.sh`, which disables the\nreporting of fatal errors to PCI-e root complex before programming and triggers\na PCI-e link re-scan after programming.  The script takes the device BDF (i.e.,\n`BB:DD.FF` without the domain, which usually is `0000`) as the single input, and\nshould be ran on the server with the FPGA card.\n\nThere are two limitations related to the script.  First, if an FPGA is not yet\nprogrammed with a PCI-e enabled bitstream, it would not have an BDF address.  In\nthis case, the script does not work.  The second case is when OpenNIC driver is\nalready loaded, it can hang the kernel after the script issues link re-scan.\nThis issue is planned to be addressed in the future driver release.\n\nFor the above two cases, the safest workaround is to use a different server to\nprogram the FPGA.  For configuration memory programming, do a cold reboot to\ntrigger the FPGA boot process.\n\nFor the \"Program the device directly\" way, consider using/referring to\n`script/program_fpga.sh`. Note, if this is the first time programming FPGA with\nthe bitstream then the server needs to be warm rebooted. No reboot is required\nfor later programming (unless a cold reboot is issued).\n\n## Known Issues\n\n### Server Boot Failure after FPGA Programming\n\nA warm reboot is needed after loading the bitstream onto the FPGA.  But this\nreboot fails with the error message: \n\n    A PCIe link training failure is observed in Slot1 and the link is disabled.\n\nFor Dell servers, there is a temporary hack [discussed\nhere](https://forums.xilinx.com/t5/PCIe-and-CPM/PCIE-link-training-error-on-DELL-R730/m-p/806428).\nThe trick is to issue a second warm reboot command using iDRAC while the system\nis rebooting and before PCIe endpoint detection.  The hypothesis is that this\ngives enough time to load the configuration on the FPGA.  This seems to be\nworking so far.\n\n### CMAC license\n\nSome users have reported this error when trying to build the shell.\n\n    ERROR: [Common 17-69] Command failed: This design contains one or more cells\n    for which bitstream generation is not permitted:\n    `cmac_subsystem_inst/cmac_wrapper_inst/cmac_inst/inst/i_cmac_usplus_0_top\n    (\u003cencrypted cellview\u003e)`.  If a new IP Core license was added, in order for\n    the new license to be picked up, the current netlist needs to be updated by\n    resetting and re-generating the IP output products before bitstream\n    generation.\n\nSince CMAC is hardened in Ultrascale+, `cmac_usplus` has a free license.  To get\nthe CMAC license, go to [www.xilinx.com/getlicense](https://www.xilinx.com/getlicense).\nAfter login, click \"Search Now\" in the \"Evaluation and No Charge Cores\" box on\nthe right side of the page.  You will see a popup with a \"Search\" box at top\nleft.  Enter \"100G\" in the search box.  You will see \"UltraScale+ Integrated\n100G Ethernet No Charge License\".  Select this and click \"Add\". Then you'll be\nable to find it in the \"Certificate Based Licenses\" table. Select it and click\n\"Generate Node-Locked License\". A screenshot could be found ![here](vivado_cmac.png).\n\n---\n\n# Copyright Notice and Disclaimer\n\nThis file contains confidential and proprietary information of Xilinx, Inc. and\nis protected under U.S. and international copyright and other intellectual\nproperty laws.\n\nDISCLAIMER\n\nThis disclaimer is not a license and does not grant any rights to the materials\ndistributed herewith.  Except as otherwise provided in a valid license issued to\nyou by Xilinx, and to the maximum extent permitted by applicable law: (1) THESE\nMATERIALS ARE MADE AVAILABLE \"AS IS\" AND WITH ALL FAULTS, AND XILINX HEREBY\nDISCLAIMS ALL WARRANTIES AND CONDITIONS, EXPRESS, IMPLIED, OR STATUTORY,\nINCLUDING BUT NOT LIMITED TO WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR\nFITNESS FOR ANY PARTICULAR PURPOSE; and (2) Xilinx shall not be liable (whether\nin contract or tort, including negligence, or under any other theory of\nliability) for any loss or damage of any kind or nature related to, arising\nunder or in connection with these materials, including for any direct, or any\nindirect, special, incidental, or consequential loss or damage (including loss\nof data, profits, goodwill, or any type of loss or damage suffered as a result\nof any action brought by a third party) even if such damage or loss was\nreasonably foreseeable or Xilinx had been advised of the possibility of the\nsame.\n\nCRITICAL APPLICATIONS\n\nXilinx products are not designed or intended to be fail-safe, or for use in any\napplication requiring failsafe performance, such as life-support or safety\ndevices or systems, Class III medical devices, nuclear facilities, applications\nrelated to the deployment of airbags, or any other applications that could lead\nto death, personal injury, or severe property or environmental damage\n(individually and collectively, \"Critical Applications\"). Customer assumes the\nsole risk and liability of any use of Xilinx products in Critical Applications,\nsubject only to applicable laws and regulations governing limitations on product\nliability.\n\nTHIS COPYRIGHT NOTICE AND DISCLAIMER MUST BE RETAINED AS PART OF THIS FILE AT\nALL TIMES.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxilinx%2Fopen-nic-shell","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fxilinx%2Fopen-nic-shell","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fxilinx%2Fopen-nic-shell/lists"}