{"id":19706062,"url":"https://github.com/llnl/pf3dk","last_synced_at":"2026-06-17T10:31:50.287Z","repository":{"id":66082978,"uuid":"298351485","full_name":"LLNL/PF3DK","owner":"LLNL","description":"PF3DK contains kernels derived from pF3D and is intended for use in testing compiler optimization. It contains CPU and GPU (OpenMP 4.5) versions of the kernels. ","archived":false,"fork":false,"pushed_at":"2020-11-23T20:18:47.000Z","size":71,"stargazers_count":1,"open_issues_count":0,"forks_count":1,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-04-12T05:03:34.620Z","etag":null,"topics":["cpu","gpu","testing"],"latest_commit_sha":null,"homepage":"","language":"C","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-3-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/LLNL.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2020-09-24T17:41:40.000Z","updated_at":"2020-12-01T19:30:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"18d50635-c65a-4334-9f52-785a3729b955","html_url":"https://github.com/LLNL/PF3DK","commit_stats":null,"previous_names":[],"tags_count":1,"template":false,"template_full_name":null,"purl":"pkg:github/LLNL/PF3DK","repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2FPF3DK","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2FPF3DK/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2FPF3DK/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2FPF3DK/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/LLNL","download_url":"https://codeload.github.com/LLNL/PF3DK/tar.gz/refs/heads/master","sbom_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/LLNL%2FPF3DK/sbom","scorecard":null,"host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":286080680,"owners_count":34445179,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2026-05-26T15:22:16.424Z","status":"online","status_checked_at":"2026-06-17T02:00:05.408Z","response_time":127,"last_error":null,"robots_txt_status":"success","robots_txt_updated_at":"2025-07-24T06:49:26.215Z","robots_txt_url":"https://github.com/robots.txt","online":true,"can_crawl_api":true,"host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["cpu","gpu","testing"],"created_at":"2024-11-11T21:33:35.275Z","updated_at":"2026-06-17T10:31:50.266Z","avatar_url":"https://github.com/LLNL.png","language":"C","funding_links":[],"categories":[],"sub_categories":[],"readme":"**PF3DK**\n=======\n\nPF3DK contains kernels extracted from pF3D, A laser-plasma simulation\ncode developed at LLNL. The kernels are intended for use in evaluating\nthe effectiveness of compiler optimizations.\n\nQuick Start\n-----------\n\nThe pF3DK repo contains build and run script for HPC systems at LLNL.\nTo get started, make a copy of a build and a run script and modify\nthem to match your system. You can type something like this to run\non an LLNL Broadwell cluster:\n\n```shell\n# build a reference code\n./build-topaz-mpi-ref.sh\n\n# build an optimized code\n./build-topaz-mpi.sh\n\n# allocate a node to run on\nsalloc -N 1 -n 1 --exclusive\n\n# This writes the results of the reference code to a text file\n./run-topaz-mpi-ref.csh\n\n# run the optimized code to measure performance and compare to\n# the reference results\n./run-topaz.csh\n```\n\nThe next section briefly describes pF3D and the kernels extracted from it.\nIf you are just interested in building and running the kernels, jump down\nto the Building and Running sections.\n\n\nOverview of pF3D\n-----------\n\npF3D is used to simulate the interaction between a high intensity laser\nand a plasma (ionized gas) in experiments performed at LLNL's National\nIgnition Facility. pF3D simulations can consume large amounts of computer\ntime, so it is important that it is well optimized.\n\npF3D uses many complex arrays and a few real arrays. These arrays\nnormally have float precision. Versions of the testsThe pF3DK kernels run\non CPUs and GPUs.\n\nPF3DK includes rotth and couple_waves kernels. These\nkernels are loosely based on functions in pF3D, but have been\nsimplified to more clearly exhibit some compiler issues that\nhave arisen during the development of pF3D.\n\nThe key points about these functions from a compiler point\nof view is that they use C99 complex variables, compute\nsines and cosines, and have float complex, float and double\nvariables. The loops are SIMDizable and have OpenMP simd directives\non CPUs. The challenges are for the compiler to recognize that a loop\ncontaining multiple data types and calls to math libraries\nis SIMDizable. For bonus points a compiler needs to figure\nout the correct SIMD size to use when the CPU supports multiple\nvector widths (hint- the goal is to make the sines and cosines\nfast). OpenMP 4.5 target offload is used by the GPU versions.\n\nPF3DK also includes some 2D FFT kernels. These kernels\nperform 2D FFTs over xy-planes. A 2D FFT is performed\nfor all xy-planes so the FFT kernel operates on 3D arrays.\n\npF3D normally runs with xy-planes decomposed into multiple\nMPI domains. Performing a 2D FFT is done by \"transposing\"\nuso that all processes have complete rows of the grid. 1D FFTs\nare performed on each row using a single process FFT.\nThe FFTs use the FFTW API or a vendor specific API\nthat is similar to FFTW (e.g. Nvidia cuFFT). Another transpose\nis used to assemble complete columns in each process. A second set\nof 1D FFTs is performed, and a final transpose takes the data\nback to the original checkerboard decomposition.\n\nThe 2D FFTs are implemented using a \"toolkit\" of functions\nthat handle data movement. The FFTs can be performed using FFTW\nor a vendor optimized FFT library. There are versions that\nrun in a single process and versions that run on multiple\nMPI processes. The use of the toolkit instead of a single\nmonolithic function makes it easier to check the performance\nof the individual pieces.\n\n\nBuilding PF3DK\n-----------\n\nThe kernels have very few external dependencies so they are easy\nto build. The compiler flag to enable OpenMP support should be set on\ncompile and link lines in the build scripts. An MPI wrapper (e.g. mpicc)\nfor the C compiler is assumed to be available. If it isn't, you will\nneed to add the appropriate include files to the compile lines and\nthe appropriate libraries to the load line.\n\nThe x86_64 build scripts are set up to use the Intel C compiler and\nlink with the Intel MKL library to access a tuned FFT. Tuned FFTs\nshould be used when available.\n\n\nRunning PF3DK\n-----------\n\nThe kernels run in a few minutes and you may  want to try them\nwith several different compiler flags in one session. The best\nway to do that at LLNL is to allocate a single node and run an\ninteractive shell on it (use salloc on a SLURM based system).\n\nThere is a script that runs a code built with very low optmization\nto generate reference values. Another script runs an optimized code\nand compares the results to the reference values.\n\nThe scripts for \"topaz\" are intended for 36 core Intel Broadwell nodes.\nThe scripts for \"ansel\" are intended for 40 core IBM Power 9 nodes\nwith 4 Nvidia Tesla GPUs. One pair of scripts checks only CPU\nperformance. The other checks both CPU and GPU performance.\n\nThe build and run scripts rely on the SYS_TYPE variable used\nat LLNL. To create your own build and run scripts, start with\nthe sample scripts and ajdust them to match your system.\n\nLicense\n-----------\n\nPF3DK is distributed under the terms of the BSD-3 license. All new contributions must be made under the BSD-3 license.\n\nSee LICENSE and NOTICE for details.\n\nSPDX-License-Identifier: BSD-3-Clause\n\nLLNL-CODE-814803\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllnl%2Fpf3dk","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fllnl%2Fpf3dk","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fllnl%2Fpf3dk/lists"}