{"id":48000484,"url":"https://github.com/zevorn/cosim-gpu","last_synced_at":"2026-04-04T12:18:28.317Z","repository":{"id":342919948,"uuid":"1175624522","full_name":"zevorn/cosim-gpu","owner":"zevorn","description":" a QEMU + gem5 co-simulation framework for AMD MI300X GPU research.","archived":false,"fork":false,"pushed_at":"2026-03-16T08:40:50.000Z","size":374,"stargazers_count":28,"open_issues_count":1,"forks_count":7,"subscribers_count":0,"default_branch":"main","last_synced_at":"2026-03-16T13:29:19.071Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":"","language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"apache-2.0","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/zevorn.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null,"notice":null,"maintainers":null,"copyright":null,"agents":null,"dco":null,"cla":null}},"created_at":"2026-03-08T00:28:53.000Z","updated_at":"2026-03-16T08:40:54.000Z","dependencies_parsed_at":null,"dependency_job_id":null,"html_url":"https://github.com/zevorn/cosim-gpu","commit_stats":null,"previous_names":["zevorn/cosim-gpu"],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/zevorn/cosim-gpu","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zevorn%2Fcosim-gpu","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zevorn%2Fcosim-gpu/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zevorn%2Fcosim-gpu/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zevorn%2Fcosim-gpu/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/zevorn","download_url":"https://codeload.github.com/zevorn/cosim-gpu/tar.gz/refs/heads/main","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/zevorn%2Fcosim-gpu/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":31399285,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-04-04T10:20:44.708Z","status":"ssl_error","status_checked_at":"2026-04-04T10:20:06.846Z","response_time":60,"last_error":"SSL_read: unexpected eof while reading","robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":false,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2026-04-04T12:18:27.813Z","updated_at":"2026-04-04T12:18:28.273Z","avatar_url":"https://github.com/zevorn.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"# QEMU + gem5 MI300X Co-simulation\n\n[中文文档](README.zh.md)\n\nCo-simulation framework that pairs **QEMU** (host CPU/system via KVM) with\n**gem5** (cycle-accurate MI300X GPU model) to run real AMD ROCm/HIP workloads\non a simulated GPU without physical hardware.\n\n```\n+-----------------------------+       +----------------------------+\n|  QEMU  (Q35 + KVM)          |       |  gem5  (Docker)            |\n|  +-----------------------+  |       |  +----------------------+  |\n|  | Guest Linux           |  |       |  | MI300X GPU Model     |  |\n|  | amdgpu driver         |  |       |  |  Shader / CU / SDMA  |  |\n|  | ROCm 7.0 / HIP        |  |       |  |  PM4 / Ruby caches   |  |\n|  +----------+------------+  |       |  +---------+------------+  |\n|  +----------v------------+  |       |  +---------v------------+  |\n|  | vfio-user-pci         |\u003c--------\u003e|  | MI300XVfioUser       |  |\n|  | (QEMU built-in)       |  |vfio-  |  | (libvfio-user)       |  |\n|  +-----------------------+  |user   |  +----------------------+  |\n+-----------------------------+       +----------------------------+\n        |                                       |\n        v                                       v\n  /dev/shm/cosim-guest-ram            /dev/shm/mi300x-vram\n  (shared guest RAM)                  (shared GPU VRAM)\n```\n\n## Features\n\n- **Full amdgpu driver load** — DRM initialized, 7 XCP partitions, gfx942\n- **HIP compute verified** — hipMalloc, kernel dispatch, hipDeviceSynchronize\n- **MSI-X interrupts** — gem5 → QEMU interrupt forwarding, IH ring buffer\n- **Shared memory DMA** — zero-copy VRAM and guest RAM via `/dev/shm`\n- **Auto driver load** — systemd service for `ip_block_mask=0x67 discovery=2`\n\n## Prerequisites\n\n| Requirement | Details |\n|---|---|\n| Host OS | Linux x86_64 with KVM (tested on WSL2 6.6.x) |\n| Docker | Running daemon, user in `docker` group |\n| KVM | `/dev/kvm` accessible |\n| Disk space | ~120 GB (55G disk image + build artifacts) |\n| RAM | 16 GB+ recommended |\n\n## Quick Start\n\n### Option A: Script-based\n\n```bash\ngit clone --recurse-submodules git@github.com:zevorn/cosim.git\ncd cosim\n\n# Build gem5 + QEMU + disk image (~2h total, needs KVM + Docker + ~60GB disk)\nGEM5_BUILD_IMAGE=ghcr.io/gem5/gpu-fs:latest ./scripts/run_mi300x_fs.sh build-all\n\n# Build runtime Docker image (for running gem5 inside Docker)\ncd scripts \u0026\u0026 docker build -t gem5-run:local -f Dockerfile.run . \u0026\u0026 cd ..\n\n# Launch co-simulation\n./scripts/cosim_launch.sh\n```\n\n### Option B: Manual step-by-step\n\n```bash\n# 1. Clone with submodules\ngit clone --recurse-submodules git@github.com:zevorn/cosim.git\ncd cosim\n\n# 2. Build gem5 (in Docker, ~30min; use -j1 if OOM-killed during linking)\ncd gem5\ndocker run --rm -v \"$(pwd):/gem5\" -w /gem5 \\\n    -e PYTHONPATH=/usr/lib/python3.12/lib-dynload \\\n    ghcr.io/gem5/gpu-fs:latest \\\n    bash -c \"scons build/VEGA_X86/gem5.opt -j4 GOLD_LINKER=True --linker=gold\"\ncd ..\n\n# 3. Build runtime Docker image (for running gem5 inside Docker)\ncd scripts \u0026\u0026 docker build -t gem5-run:local -f Dockerfile.run . \u0026\u0026 cd ..\n\n# 4. Build QEMU (stock build; vfio-user-pci is built-in since QEMU 10.0)\ncd qemu \u0026\u0026 mkdir -p build \u0026\u0026 cd build\n../configure --target-list=x86_64-softmmu\nmake -j$(nproc)\ncd ../..\n\n# 5. Pre-build m5 utility (recommended — avoids git clone inside guest VM)\ndocker run --rm -v \"$(pwd)/gem5:/gem5\" -w /gem5 \\\n    ghcr.io/gem5/gpu-fs:latest \\\n    bash -c \"cd util/m5 \u0026\u0026 scons build/x86/out/m5\"\ncp gem5/util/m5/build/x86/out/m5 gem5-resources/src/x86-ubuntu-gpu-ml/files/\n\n# 6. Build disk image (Ubuntu 24.04 + ROCm 7.0, ~40min, needs KVM + ~60GB disk)\n./scripts/run_mi300x_fs.sh build-disk\n\n# 7. Launch co-simulation\n./scripts/cosim_launch.sh\n```\n\nAfter guest boots (auto-login as root), the GPU driver loads automatically\nvia `cosim-gpu-setup.service` (~40s). Verify:\n\n```bash\nrocm-smi          # should show device 0x74a0\nrocminfo          # should show gfx942\n\n# Manual setup (if the systemd service is not installed):\ndd if=/root/roms/mi300.rom of=/dev/mem bs=1k seek=768 count=128\nmodprobe amdgpu ip_block_mask=0x67 discovery=2 ras_enable=0\n\n# Run a HIP test\ncat \u003e /tmp/test.cpp \u003c\u003c 'EOF'\n#include \u003chip/hip_runtime.h\u003e\n#include \u003ccstdio\u003e\n__global__ void add(int *a, int *b, int *c, int n) {\n    int i = blockIdx.x * blockDim.x + threadIdx.x;\n    if (i \u003c n) c[i] = a[i] + b[i];\n}\nint main() {\n    const int N = 4;\n    int ha[] = {1,2,3,4}, hb[] = {10,20,30,40}, hc[4] = {};\n    int *da, *db, *dc;\n    hipMalloc(\u0026da, N*4); hipMalloc(\u0026db, N*4); hipMalloc(\u0026dc, N*4);\n    hipMemcpy(da, ha, N*4, hipMemcpyHostToDevice);\n    hipMemcpy(db, hb, N*4, hipMemcpyHostToDevice);\n    add\u003c\u003c\u003c1, N\u003e\u003e\u003e(da, db, dc, N);\n    hipMemcpy(hc, dc, N*4, hipMemcpyDeviceToHost);\n    printf(\"Result: %d %d %d %d\\n\", hc[0], hc[1], hc[2], hc[3]);\n    printf(\"%s\\n\", (hc[0]==11\u0026\u0026hc[1]==22\u0026\u0026hc[2]==33\u0026\u0026hc[3]==44) ? \"PASSED!\" : \"FAILED!\");\n    hipFree(da); hipFree(db); hipFree(dc);\n}\nEOF\n/opt/rocm/bin/hipcc --offload-arch=gfx942 -o /tmp/test /tmp/test.cpp\n/tmp/test\n# Expected: Result: 11 22 33 44\n#           PASSED!\n```\n\n## Repository Structure\n\n```\ncosim/\n|-- gem5/                    # gem5 simulator (submodule, cosim-gpu branch)\n|   |-- src/dev/amdgpu/      # MI300X GPU device model \u0026 vfio-user bridge\n|   |-- ext/libvfio-user/    # libvfio-user library (Nutanix)\n|   `-- configs/example/gpufs/mi300_cosim.py  # cosim configuration\n|-- qemu/                    # QEMU emulator (submodule, stock QEMU 10.0+)\n|-- gem5-resources/          # disk images, kernels, GPU apps (submodule)\n|-- scripts/                 # build \u0026 launch scripts\n|   |-- cosim_launch.sh      # one-click cosim launcher\n|   |-- run_mi300x_fs.sh     # build orchestration\n|   |-- cosim_guest_setup.sh # guest-side GPU setup\n|   |-- cosim_test_client.py # socket test client\n|   `-- Dockerfile.run       # gem5 runtime Docker image\n|-- docs/                    # technical documentation (zh + en)\n|   |-- en/                  # English\n|   `-- zh/                  # Chinese\n|-- LICENSE                  # Apache 2.0\n`-- README.md                # this file\n```\n\n## Key Components\n\n### QEMU Side (stock `vfio-user-pci`)\n\nUses QEMU's built-in `vfio-user-pci` device (no custom QEMU code). QEMU connects\nto gem5's vfio-user server and maps all BARs through the standard vfio-user protocol:\n- **BAR0+1**: VRAM (64-bit, prefetchable, shared memory backed)\n- **BAR2+3**: Doorbell (64-bit, forwarded to gem5 via vfio-user)\n- **BAR4**: MSI-X (256 vectors, eventfd-based KVM injection)\n- **BAR5**: MMIO registers (32-bit, forwarded to gem5 via vfio-user)\n\n### gem5 Side (`gem5/src/dev/amdgpu/`)\n\n- **MI300XVfioUser** — vfio-user server (libvfio-user), BAR/config/IRQ dispatch\n- **AMDGPUDevice** — MI300X GPU device model (MMIO, doorbell, config space)\n- **PM4PacketProcessor** — command processor with VRAM-aware fence routing\n- **SDMAEngine** — DMA engine with VRAM write-back support\n- **AMDGPUVM** — GART translation with cosim fallback (shared VRAM PTE reads)\n\n## Version Matrix\n\n| Component | Version |\n|---|---|\n| Guest OS | Ubuntu 24.04.2 LTS |\n| Guest kernel | 6.8.0-79-generic |\n| ROCm | 7.0.0 |\n| GPU device | MI300X (gfx942, DeviceID 0x74A0) |\n| gem5 build | VEGA_X86, GPU_VIPER coherence |\n\n## Documentation\n\nDetailed technical documentation is available in [`docs/`](docs/):\n\n- [Complete Usage Guide](docs/en/cosim-usage-guide.md) — build, run, test\n- [Technical Notes](docs/en/cosim-technical-notes.md) — architecture, pitfalls, fixes\n- [MI300X Memory Management](docs/en/mi300x-memory-management.md) — GART, address translation\n- [GPU FS Guide](docs/en/gpu-fs-guide.md) — gem5 standalone GPU full-system simulation\n- [Guest GPU Init](docs/en/cosim-guest-gpu-init.md) — driver initialization flow\n- [Memory Architecture](docs/en/cosim-memory-architecture.md) — shared memory, VRAM routing, DMA\n- [Debugging Pitfalls](docs/en/cosim-debugging-pitfalls.md) — common issues and solutions\n- [Development Story](docs/en/cosim-dev-story.md) — how this project was built in one day with Claude\n\n## License\n\nThis project is licensed under the [Apache License 2.0](LICENSE).\n\nNote: The `gem5` and `qemu` submodules are governed by their own respective\nlicenses (gem5: BSD-3-Clause, QEMU: GPL-2.0).\n\n## Acknowledgments\n\n- [gem5](https://www.gem5.org/) — modular computer architecture simulator\n- [QEMU](https://www.qemu.org/) — open-source machine emulator\n- [ROCm](https://rocm.docs.amd.com/) — AMD GPU computing platform\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzevorn%2Fcosim-gpu","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fzevorn%2Fcosim-gpu","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fzevorn%2Fcosim-gpu/lists"}