{"id":19867737,"url":"https://github.com/cwsmith/epyc_vs_skylake","last_synced_at":"2026-06-09T08:32:20.531Z","repository":{"id":93888988,"uuid":"134563766","full_name":"cwsmith/epyc_vs_skylake","owner":"cwsmith","description":"Testing code performance on AMD EPYC and Intel Skylake","archived":false,"fork":false,"pushed_at":"2018-06-21T14:40:44.000Z","size":5498,"stargazers_count":3,"open_issues_count":0,"forks_count":0,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-03-01T00:39:15.025Z","etag":null,"topics":[],"latest_commit_sha":null,"homepage":null,"language":"Shell","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/cwsmith.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2018-05-23T12:06:53.000Z","updated_at":"2023-12-21T20:47:09.000Z","dependencies_parsed_at":"2023-04-26T15:02:05.142Z","dependency_job_id":null,"html_url":"https://github.com/cwsmith/epyc_vs_skylake","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"purl":"pkg:github/cwsmith/epyc_vs_skylake","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cwsmith%2Fepyc_vs_skylake","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cwsmith%2Fepyc_vs_skylake/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cwsmith%2Fepyc_vs_skylake/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cwsmith%2Fepyc_vs_skylake/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/cwsmith","download_url":"https://codeload.github.com/cwsmith/epyc_vs_skylake/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/cwsmith%2Fepyc_vs_skylake/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34098931,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-09T02:00:06.510Z","response_time":63,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":[],"created_at":"2024-11-12T15:30:12.510Z","updated_at":"2026-06-09T08:32:20.512Z","avatar_url":"https://github.com/cwsmith.png","language":"Shell","funding_links":[],"categories":[],"sub_categories":[],"readme":"## MFEM \n\nClone the benchmarking repo\n\n`git clone git@github.com:CEED/benchmarks.git ceed-benchmarks`\n\nThe scripts provided by this repo with download and build mfem, and then run the\n`bp1` benchmark on a range of processes with varying basis function order\nand mesh size.\n\n`bp1` requires the total number of processes be a power of two.  Thus, on both\nSkylake and EPYC tests do not use all available cores/hardware threads.\n\n### Stampede2\n\n`bp1` was executed on a dual socket Intel Xeon Platinum 8160 Skylake node using\nGCC 7.1.0 with Intel MPI 17 and Intel 18.0.0 with Intel MPI 18.\n\nThe [Stampede2 User Guide provides the following information on the Skylake nodes](\nhttps://portal.tacc.utexas.edu/user-guides/stampede2#overview-skxcomputenodes)\n```\nModel:  Intel Xeon Platinum 8160 (\"Skylake\")\nTotal cores per SKX node:   48 cores on two sockets (24 cores/socket)\nHardware threads per core:  2\nHardware threads per node:  48 x 2 = 96\nClock rate:   2.1GHz nominal (1.4-3.7GHz depending on instruction set and number of active cores)\nRAM:  192GB (2.67GHz)\nCache:  32KB L1 data cache per core; 1MB L2 per core; 33MB L3 per socket. Each\nsocket can cache up to 57MB (sum of L2 and L3 capacity).\n```\n\nhttps://ark.intel.com/products/120501/Intel-Xeon-Platinum-8160-Processor-33M-Cache-2_10-GHz\n\nRun the tests:\n\n```\ncp epyc_vs_skylake/mfem/machine-configs/stampede2.sh ceed-benchmarks/machine-configs\nidev -N1 -p skx-dev\ncd ceed-benchmarks/tests/mfem_bps/\n../../go.sh -c stampede2 -m intel -r bp1_v1.sh -n \"4 8 16 32\" -p \"4 8 16 32\" \u0026\u003e stampede2_bp1_intel18.txt\n../../go.sh -c stampede2 -m gcc -r bp1_v1.sh -n \"4 8 16 32\" -p \"4 8 16 32\" \u0026\u003e stampede2_bp1_gcc7.txt\n```\n\n### IPA\n\n`bp1` was executed on a dual socket AMD EPYC 7451 node using GCC 7.1.0 and AOCC 1.2 \nwith MVAPICH2, and the 2018 Intel compiler and MPI implementation.\nEach 7451 has 24 cores and supports two hardware threads per core (96 threads in\ntotal).  The node has 256GB (16GB x16) of RAM running at 2.67Ghz.\n\nhttp://www.cpu-world.com/CPUs/Zen/AMD-EPYC%207451.html\n\nThe AOCC 1.2 C++ and Fortran (not required for mfem) was installed using the\ntarballs and directions here:\n\nhttps://developer.amd.com/amd-aocc/\n\nRun the tests:\n\n```\ncp epyc_vs_skylake/mfem/machine-configs/ipa.sh ceed-benchmarks/machine-configs\nsalloc -N 1 -t60 -p epyc\ncd ceed-benchmarks/tests/mfem_bps/\n../../go.sh -c ipa -m aocc -r bp1_v1.sh -n \"4 8 16 32\" -p \"4 8 16 32\" \u0026\u003e ipa_bp1_aocc12.txt\n../../go.sh -c ipa -m gcc -r bp1_v1.sh -n \"4 8 16 32\" -p \"4 8 16 32\" \u0026\u003e ipa_bp1_gcc7.txt\n```\n\n### Lyra\n\n`bp1` was executed on a single socket AMD EPYC 7451 node using AOCC 1.2 with\nMPICH 3.1.2.  Each 7451 has 24 cores and supports two hardware threads per core (48 threads in\ntotal).  The node has 128GB of RAM running at 2.67Ghz.  Note, the motherboard\ndesign places two DIMMs per channel ('dual rank') which results in the memory\nonly running at 2.4Ghz; a 10% reduction.\n\nThe AOCC 1.2 C++ and Fortran (not required for mfem) was installed using the\ntarballs and directions here:\n\nhttps://developer.amd.com/amd-aocc/\n\nRun the tests:\n\n```\ncp epyc_vs_skylake/mfem/machine-configs/lyra.sh ceed-benchmarks/machine-configs\nsalloc -N 1 -t60\ncd ceed-benchmarks/tests/mfem_bps/\nprocs=\"3 6 12 24 48\"\nbase_nxyz=\"1 1 3\" ../../go.sh -c lyra -m aocc -r bp1_v1.sh -n \"$procs\" -p \"$procs\" \u0026\u003e lyra_bp1_aocc12.txt\n```\n\n### Post Processing\n\nGenerate plots for each system-compiler pair:\n\n```\ncd epyc_vs_skylake/mfem/bp1/\u003csystem-compiler\u003e/\nln -s ceed-benchmarks/tests/mfem_bps/*.py .\npython postprocess-plot-1.py *.txt\npython postprocess-plot-2.py *.txt\npython postprocess-plot-3.py *.txt\n```\n\nGenerate comparison plots between system-compiler pairs A and B:\n\n```\ncd epyc_vs_skylake/mfem/bp1/\u003csystem-compiler-A_vs_system-compiler-B\u003e/\nln -s ceed-benchmarks/tests/mfem_bps/*.py .\npython postprocess-plot-4.py ../\u003csystem-compiler-A\u003e/*.txt ../\u003csystem-compiler-B\u003e/*.txt\n```\n\n\n## STREAM Triad\n\n### Stampede2\n\nPeak OpenMP STREAM Triad performance using the Intel18 compiler on the two socket Skylake nodes on Stampede2 is 211 GB/s.\n\nThe following TACC benchmarking report lists a 194 GB/s using the Intel17 compiler:\n\nhttps://repositories.lib.utexas.edu/bitstream/handle/2152/61472/SKX_Benchmarking.pdf?sequence=2\u0026isAllowed=y\n\n.  Colfax Research also has an article here the discusses other Xeon SKUs:\n\nhttps://colfaxresearch.com/xeon-2017/#sec-3\n\n### IPA\n\nPeak OpenMP STREAM Triad performance on the two socket EPYC nodes on IPA is 200 GB/s\nusing GCC 7.1.0.  \n\nAOCC 1.2 test results peak at 153 GB/s and oscillate significantly versus thread count.\n\nThe following AnAndTech and Tirias Research articles discuss the EPYC NUMA domains.  AnAndTech testing produces a slightly higher peak bandwidth of 207 GB/s using the Intel compilers.\n\nhttps://www.anandtech.com/show/11544/intel-skylake-ep-vs-amd-epyc-7000-cpu-battle-of-the-decade/12\n\nhttps://www.amd.com/system/files/2018-03/AMD-Optimizes-EPYC-Memory-With-NUMA.pdf\n\n#### execution\n\nAllocate an EPYC node and run the `runStream` script.\n\n```\n./runStream.sh \u003cdownload=0|1\u003e \u003ccompiler=gcc|aocc\u003e\n```\n\n### Lyra\n\nPeak OpenMP STREAM Triad performance on the one socket EPYC nodes on Lyra is 101 GB/s\nusing GCC 5.4.0.  \n\nAOCC 1.2 test results peak at 85 GB/s and oscillate significantly versus thread count.\n\n#### execution\n\nAllocate an EPYC node and run the `runStream` script.\n\n```\n./runStream.sh \u003cdownload=0|1\u003e \u003ccompiler=gcc|aocc\u003e\n```\n\n\n\n### Post Processing\n\nThis produces \\*.log and \\*.log.triad files with the STREAM output and the triad\npeak bandwidth results, respectively.  The contents of the \\*.log.triad files\ncan be plotted (generates `\u003coutfig name\u003e.png`) with the `plot.py` script:\n\n```\n../plot.py  \u003coutfig name\u003e \u003cinput log\u003e [...\u003cinput log\u003e]\n```\n\n\n\n\n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcwsmith%2Fepyc_vs_skylake","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fcwsmith%2Fepyc_vs_skylake","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fcwsmith%2Fepyc_vs_skylake/lists"}