Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/feltor-dev/feltor

Numerical methods for edge and scrape-off layer blob and turbulence simulations. Homepage:
https://github.com/feltor-dev/feltor

c-plus-plus computational-fluid-dynamics computational-physics cuda discontinuous-galerkin gpu mpi numerical-methods numerical-simulations openmp parallel-algorithm parallel-computing plasma-physics plasma-turbulence

Last synced: about 1 month ago
JSON representation

Numerical methods for edge and scrape-off layer blob and turbulence simulations. Homepage:

Awesome Lists containing this project

README 

          
= Welcome to the FELTOR project!

:source-highlighter: pygments

:toc: macro



Visit our project https://feltor-dev.github.io[Homepage] for

documentations, user-guides, examples and more!



image::3dpic.jpg[3dsimulation]



FELTOR (Full-F ELectromagnetic code in TORoidal geometry) is a modular scientific software package used for:



- Physics: study fluid models for magnetised (fusion) plasmas in one, two and three dimensions

- Numerics: develop and study numerical methods for these models in particular novel discontinuous Galerkin methods and structured grid generators

- High performance computing: investigate parallel peformance, binary reproducibility and accuracy of the above algorithms on modern hardware architectures.



FELTOR applications are platform independent and run on a large variety of hardware from laptop CPUs to GPUs to high performance compute clusters.



https://zenodo.org/badge/latestdoi/14143578[image:https://zenodo.org/badge/14143578.svg[DOI]]

link:LICENSE[image:https://img.shields.io/badge/License-MIT-yellow.svg[License:

MIT]]

https://github.com/feltor-dev/feltor/releases/latest[image:https://img.shields.io/github/v/release/feltor-dev/feltor[GitHub release (latest by date)]]



toc::[]



== 1. Quick start guide [[sec_quickstart]]

This guide discusses how to setup, build, test and benchmark FELTOR on a given system. Please read it before you proceed to the https://feltor-dev.github.io/user-guide[user guide] to learn how to use the library in your own programs.



=== System Setup using CMake

The first step is to clone and configure the FELTOR repository from GitHub.

[source,sh]

----

git clone https://www.github.com/feltor-dev/feltor

cd feltor

cmake --preset cpu # or gpu, omp, mpi-cpu, mpi-gpu, mpi-omp

----

You may need to install the external dependencies `libnetcdf-dev`, `liblapack-dev` and `libboost-dev` (and `libglfw3-dev` for OpenGL output and `libopenmpi-dev` for MPI support) from your system package manager.

____

On Windows you can use the built-in git client and cmake integration of Visual Studio. It is easiest to use the built-in vcpkg manager to install the `netcdf-c`, `glfw3`, `lapack` and `boost-headers` dependencies.

____



There are 6 presets targeting 6 different hardware architectures. All require the C{plus}{plus}-17 standard. Each preset `X` creates its binary directory `build/X`.



.System requirements [[tab_requirements]]

[cols='3,10,14',options="header"]

|=======================================================================

| CMake Preset  | Requirements | Description

| *cpu*     | gcc >= 9 or msvc >= 19 or icc >= 19.0 or clang >= 19        |Single core CPU, no parallelization; support for AVX and FMA instruction set is recommended

| *omp* | OpenMP >= 2 | Multi-core CPU parallelisation, set OMP_NUM_THREADS environment variable to set number of OpenMP threads to run.

| *gpu*    |   nvcc >= 11.0 | Parallel computation on a single NVidia GPU

| *mpi-cpu*     | MPI >= 3 | Distributed memory system for pure MPI parallelisation

| *mpi-omp*| MPI >= 3, OpenMP >= 2 | Hybrid MPI + OpenMP parallelisation. In this configuration you may want to investigate how OpenMP threads map to CPU cores.

| *mpi-gpu*| MPI >= 3, nvcc >= 11.0| Hybrid MPI + GPU parallelisation. Each MPI thread targets one GPU. The MPI implementation should ideally be CUDA aware.

|=======================================================================



____

Our GPU backend uses the

https://developer.nvidia.com/cuda-zone[Nvidia-CUDA] programming

environment and in order to compile and run a program for a GPU a user

needs the nvcc compiler and a NVidia

GPU. However, we explicitly note here that due to the modular design of

our software a user does not have to possess a GPU nor the nvcc

compiler. The CPU version of the backend is equally valid and provides

the same functionality. Analogously, an MPI installation is only required if the user targets

a distributed memory system.

____

=== Available targets

The Feltor projects defines a host of CMake targets that can be

built after configuration. To build everything run

[source,sh]

----

cmake --build build/cpu -j 4

 # Replace "cpu" with the preset of your choice; here and in the following.

 # -j 4 activates parallel compilation with 4 threads.

----

The Feltor CMake targets are organised into three categories: tests, benchmarks and production projects. These

can be targeted individually using

[source,sh]

----

cmake --build build/cpu --target dg_tests -j 4

# Compile all tests of the dg library

cmake --build build/cpu --target dg_benchmarks -j 4

# Compile all benchmarks of the dg library

cmake --build build/cpu --target feltor_projects -j 4

# Compile all production projects of feltor

----

The tests and benchmarks will be built in `build/X/inc` and the projects in `build/X/src`, where `X` is the preset in use. The location of the executables in `build/X` exactly mirror the folder structure of the original C++ files in `feltor`, for example `feltor/inc/dg/blas_b.cpp` compiles to `build/X/inc/dg/blas_b`.

The tests can be run using

[source,sh]

----

ctest --test-dir build/cpu

----



Lastly, one can also target individual programs. All programs in the dg library start with the prefix `dg_` followed by the component name `backend`, `topology`, `geometries`, `file` or `matrix` followed by the program name without suffix. The feltor project targets follow the naming scheme `project_target` where `project` is the name of the folder in the `src` directory and `target` is the program name without suffix. Again, the output name in the binary directory follows the original folder structure and program name. For example:

[source,sh]

----

cmake --build build/cpu --target dg_blas_b

./build/cpu/inc/dg/blas_b

# Compile and run the benchmark program feltor/inc/dg/blas_b.cpp

cmake --build build/cpu --target dg_topology_derivatives_t

./build/cpu/inc/dg/topology/derivatives_t

# Compile and run the test program feltor/inc/dg/topology/derivatives_t.cpp

cmake --build build/cpu --target feltor_feltor

./build/cpu/src/feltor/feltor

# Compile and run the 3d feltor code in feltor/src/feltor/feltor.cpp

----

Again, remember to replace `cpu` with the preset of your choice and mind the various options when running parallel programs, e.g.

[source,sh]

----

cmake --preset gpu

cmake --build build/gpu --target dg_blas_b

./build/gpu/inc/dg/blas_b

# Compile and run the benchmark program feltor/inc/dg/blas_b.cpp for GPU

cmake --preset omp

cmake --build build/omp --target dg_blas_b

export OMP_NUM_THREADS=4

./build/gpu/inc/dg/blas_b

# Compile and run the benchmark program feltor/inc/dg/blas_b.cpp for OpenMP

cmake --preset mpi-cpu

cmake --build build/mpi-cpu --target feltor_feltor

mpirun -n 4 ./build/mpi-cpu/src/feltor/feltor

# Compile and run the 3d feltor code in feltor/src/feltor/feltor.cpp for pure MPI using 4 MPI threads

----

=== Using FELTOR's dg library in CMake



FELTOR contains a library called the *dg-library* (from discontinuous Galerkin). To integrate FELTOR's dg library in your own project via cmake currently the only option is to add it as a submodule i.e. either (i) use https://cmake.org/cmake/help/latest/module/FetchContent.html[FetchContent] directly or (ii) use the cmake package manager https://github.com/cpm-cmake/CPM.cmake[CPM] (our recommendation) or (iii) add feltor as a git submodule and use `add_subdirectory` in your `CMakeLists.txt`. We here show the CPM version. To get started follow the CPM quick start guide to setup the file `cmake/CPM.cmake`. It is also highly recommended to set the `CPM_SOURCE_CACHE` environment variable.

____

CMake's install rules and `find_package` currently does not work well with targets that can be compiled for various languages (see https://github.com/feltor-dev/feltor/issues/59[this issue])

____

The available library targets in cmake are of the format `feltor::dg::component`, where `component` is one of the following:



.Feltor's dg library targets `feltor::dg::component`

[cols='3,10,14',options="header"]

|=======================================================================

| component  | Corresponding Header | Description

| `dg`    | `dg/algorithm.h`        |Depends on https://github.com/NVIDIA/cccl[cccl] and https://github.com/vectorclass/version2[vectorclass] (loaded via `CPMAddPackage`)

| `geometries` | `dg/geometries/geometries.h`| Depends on `feltor::dg::file::json`

| `matrix`   |  `dg/matrix/matrix.h` | Depends on `liblapack-dev` and `libboost-dev`

| `file`    | `dg/file/file.h` | Depends on `feltor::dg::file::json` and `feltor::dg::file::netcdf`

| `file::json`| `dg/file/json_utilities.h` | Depends on either `nlohmann_json >= 3.11` (default) or `jsoncpp >= 1.9.5` (setting `FELTOR_FILE_WITH_JSONCPP ON`) via `CPMAddPackage`

| `file::netcdf`| `dg/file/nc_utilities.h`| Depends on `libnetcdf-dev`.

|=======================================================================



____

As noted before you may need to install the external dependencies `libnetcdf-dev`, `liblapack-dev` and `libboost-dev` from your system package manager (or use e.g. the vcpkg manager to install  `netcdf-c`, `lapack` and `boost-headers`). Note that you can set the options `FELTOR_DG_WITH_MATRIX OFF` and `FELTOR_FILE_WITH_NETCDF OFF` to avoid having to install netcdf, lapack or boost.

____



Furthermore, since feltor's dg library depends on cccl, we inherit their option `CCCL_THRUST_DEVICE_SYSTEM`, which can be either `CPP`, `OMP` or `CUDA`. Since with CUDA a new language must be enabled (which can only be done once in a cmake project) we must add this to the cmake file:



.CMakeLists.txt

[source,cmake]

----

cmake_minimum_required(VERSION 3.26)

project( myProject

    VERSION         1.0.0

    LANGUAGES       CXX

)

# We need to enable CUDA language if the user wants it

if(CCCL_THRUST_DEVICE_SYSTEM STREQUAL "CUDA" OR CCCL_THRUST_DEVICE_SYSTEM STREQUAL "")

    enable_language(CUDA)

    set_source_files_properties(main.cpp PROPERTIES LANGUAGE CUDA)

endif()



include(cmake/CPM)



CPMAddPackage(

    NAME feltor

    GITHUB_REPOSITORY "feltor-dev/feltor"

    VERSION 8.2

    SYSTEM ON

    EXCLUDE_FROM_ALL ON

    OPTIONS "FELTOR_DG_WITH_MATRIX OFF" "FELTOR_FILE_WITH_NETCDF OFF"

)

add_executable(main main.cpp)

# The base dg library header "dg/algorithm.h"

target_link_libraries( main PRIVATE feltor::dg::dg)

----



Note

that the dg library is **header-only**, which means that you just have to

include the relevant header(s) and you're good to go. For example in the

following program we compute the square L2 norm of a

function:



.main.cpp [[main_cpp]]

[source,c++]

----

#include 

//include the basic dg-library

#include "dg/algorithm.h"



double function(double x, double y){return exp(x)*exp(y);}

int main()

{

    //create a 2d discretization of [0,2]x[0,2] with 3 polynomial coefficients

    dg::CartesianGrid2d g2d( 0, 2, 0, 2, 3, 20, 20);

    //discretize a function on this grid

    const dg::DVec x = dg::evaluate( function, g2d);

    //create the volume element

    const dg::DVec vol2d = dg::create::volume( g2d);

    //compute the square L2 norm on the device

    double norm = dg::blas2::dot( x, vol2d, x);

    // norm is now: (exp(4)-exp(0))^2/4

    std::cout << norm <

#ifdef WITH_MPI

//activate MPI in FELTOR

#include "mpi.h"

#endif

#include "dg/algorithm.h"



double function(double x, double y){return exp(x)*exp(y);}

int main(int argc, char* argv[])

{

#ifdef WITH_MPI

    //init MPI and create a 2d Cartesian Communicator assuming 4 MPI threads

    MPI_Init( &argc, &argv);

    int periods[2] = {true, true}, np[2] = {2,2};

    MPI_Comm comm;

    MPI_Cart_create( MPI_COMM_WORLD, 2, np, periods, true, &comm);

#endif

    //create a 2d discretization of [0,2]x[0,2] with 3 polynomial coefficients

    dg::CartesianMPIGrid2d g2d( 0, 2, 0, 2, 3, 20, 20

#ifdef WITH_MPI

    , comm

#endif

    );

    //discretize a function on this grid

    const dg::x::DVec x = dg::evaluate( function, g2d);

    //create the volume element

    const dg::x::DVec vol2d = dg::create::volume( g2d);

    //compute the square L2 norm

    double norm = dg::blas2::dot( x, vol2d, x);

    //on every thread norm is now: (exp(4)-exp(0))^2/4

#ifdef WITH_MPI

    //be a good MPI citizen and clean up

    MPI_Finalize();

#endif

    return 0;

}

----

The CMake file needs to be modified like



.CMakeLists.txt

[source,cmake]

----

option(MAIN_WITH_MPI "Compile main with MPI parallelisation" OFF)

if(MAIN_WITH_MPI)

    target_link_libraries(main PRIVATE MPI::MPI_CXX)

    target_compile_definitions(main PRIVATE WITH_MPI)

endif()

----



Compile e.g. for a hybrid MPI {plus} OpenMP hardware platform with



[source,sh]

----

cmake -Bbuild/mpi-omp -DCCCL_THRUST_DEVICE_SYTEM="OMP" -DCMAKE_CXX_FLAGS="-march=native -O3" -DMAIN_WITH_MPI=ON

cmake --build build/mpi-omp

export OMP_NUM_THREADS=2

mpirun -n 4 ./build/mpi-omp/main

----

This will run 4 MPI threads with 2 OpenMP threads each.



Note the striking similarity to the previous program. Especially the

line calling the dot function did not change at all. The compiler

chooses the correct implementation for you! This is a first example of __platform independent code__.



=== Using Makefiles (Deprecated)



Open a terminal and clone the repository into any folder you like



[source,sh]

----

git clone https://www.github.com/feltor-dev/feltor

----



You also need to clone https://github.com/nvidia/cccl[cccl]

 distributed under the

Apache-2.0 license. Also, we need Agner Fog's https://github.com/vectorclass/version1[vcl] library (Apache 2.0). So again in a folder of your choice



[source,sh]

----

git clone https://www.github.com/nvidia/cccl

git clone https://www.github.com/vectorclass/version2 vcl

----



____

Our code only depends on external libraries that are themselves openly

available.

If version2 of the vectorclass library does not work for you, you can also try version1.

____



==== Running a FELTOR test or benchmark program



In order to compile one of the many test and benchmark codes

inside the FELTOR library you need to tell

the FELTOR configuration where the external libraries are located on

your computer. The default way to do this is to go into your `HOME`

directory, make an include directory and link the paths in this

directory



[source,sh]

----

cd ~

mkdir include

cd include

ln -s path/to/cccl/thrust/thrust # Yes, thrust is there twice!

ln -s path/to/cccl/cub/cub

ln -s path/to/cccl/libcudacxx/include/cuda

ln -s path/to/cccl/libcudacxx/include/nv

ln -s path/to/vcl

----



____

If you do not like this, you can also set the include paths in your own config file as

described link:config/README.md[here].

____



Now let us compile the first benchmark program.



[source,sh]

----

cd path/to/feltor/inc/dg



make blas_b device=cpu #(for a single thread CPU version)

#or

make blas_b device=omp #(for an OpenMP version)

#or

make blas_b device=gpu #(if you have a GPU and nvcc )

----



Run the code with



[source,sh]

----

./blas_b

----



and when prompted for input vector sizes type for example `3 100 100 10`

which makes a grid with 3 polynomial coefficients, 100 cells in x, 100

cells in y and 10 in z. If you compiled for OpenMP, you can set the

number of threads with e.g. `export OMP_NUM_THREADS=4`.

____

This is a

benchmark program to benchmark various elemental functions the library

is built on. Go ahead and vary the input parameters and see how your

hardware performs. You can compile and run any other program that ends

in `_t.cu` (test programs) or `_b.cu` (benchmark programs) in

`feltor/inc/dg` in this way.

____



Now, let us test the mpi setup

____

You can of course skip this if you

don't have mpi installed on your computer. If you intend to use the

MPI backend, an implementation library of the mpi standard is required.

Per default `mpic++` is used for compilation.

____



[source,sh]

----

cd path/to/feltor/inc/dg



make blas_mpib device=cpu  # (for MPI+CPU)

# or

make blas_mpib device=omp  # (for MPI+OpenMP)

# or

make blas_mpib device=gpu # (for MPI+GPU, requires CUDA-aware MPI installation)

----



Run the code with `$ mpirun -n '# of procs' ./blas_mpib` then tell how

many process you want to use in the x-, y- and z- direction, for

example: `2 2 1` (i.e. 2 procs in x, 2 procs in y and 1 in z; total

number of procs is 4) when prompted for input vector sizes type for

example `3 100 100 10` (number of cells divided by number of procs must

be an integer number). If you compiled for MPI{plus}OpenMP, you can set the

number of OpenMP threads with e.g. `export OMP_NUM_THREADS=2`.



==== Running a FELTOR simulation



Now, we want to compile and run a simulation program. To this end, we have to

download and install some additional libraries for I/O-operations.



First, we need to install jsoncpp (distributed under the MIT License),

which on linux is available as `libjsoncpp-dev` through the package managment system.

For a manual build check the instructions on https://www.github.com/open-source-parsers/jsoncpp[JsonCpp].

[source,sh]

----

# You may have to manually link the include path

cd ~/include

ln -s /usr/include/jsoncpp/json

----



For data output we use the

http://www.unidata.ucar.edu/software/netcdf/[NetCDF-C] library under an

MIT - like license (we use the netcdf-4 file format).

The underlying https://www.hdfgroup.org/HDF5/[HDF5]

library also uses a very permissive license.

Both can be installed easily on Linux through the `libnetcdf-dev` and `libhdf5-dev` packages.

For a manual build follow the build instructions in the https://www.unidata.ucar.edu/software/netcdf/docs/getting_and_building_netcdf.html[netcdf-documentation].

Note that by default we use the serial netcdf and hdf5 libraries alson in the mpi

versions of applications.



Some desktop applications in FELTOR use the

https://github.com/mwiesenberger/draw[draw library] (developed by us

also under MIT), which depends on

http://www.glfw.org[glfw3], an OpenGL development library under a

BSD-like license. There is a `libglfw3-dev` package for convenient installation. Again, link `path/to/draw` in the `include` folder.



____

If you are on a HPC cluster, you may need to set INCLUDE and LIB variables manually.

For details on how FELTOR's Makefiles are configured please see the link:config/README.md[config] file. There are also examples of some existing Makefiles in the same folder.

____



We are now ready to compile and run a simulation program



[source,sh]

----

cd path/to/feltor/src/toefl # or any other project in the src folder



make toefl device=gpu     # (compile for gpu, cpu or omp)

cp input/default.json inputfile.json # create an inputfile

./toefl inputfile.json    # (behold a live simulation with glfw output on screen)

# or

make toefl_hpc device=gpu  # (compile for gpu, cpu or omp)

cp input/default_hpc.json inputfile_hpc.json # create an inputfile

./toefl_hpc inputfile_hpc.json outputfile.nc # (a single node simulation with output stored in a file)

# or

make toefl_mpi device=omp  # (compile for gpu, cpu or omp)

export OMP_NUM_THREADS=2   # (set OpenMP thread number to 1 for pure MPI)

echo 2 2 | mpirun -n 4 ./toefl_mpi inputfile_hpc.json outputfile.nc

# (a multi node simulation with now in total 8 threads with output stored in a file)

# The mpi program will wait for you to type the number of processes in x and y direction before

# running. That is why the echo is there.

----



Default input files are located in `path/to/feltor/src/toefl/input`. All

three programs solve the same equations. The technical documentation on

what equations are discretized, input/output parameters, etc. can be

generated as a pdf with `make doc` in the `path/to/feltor/src/toefl`

directory.



== 2. Documentation



The

https://mwiesenberger.github.io/feltor/dg/html/topics.html[documentation]

of the dg library was generated with

http://www.doxygen.org[Doxygen]. You can generate a local

version directly from source code. This depends on the `doxygen`,

`libjs-mathjax`, `graphviz` and `doxygen-awesome` packages. Type `make doc` in

the folder `path/to/feltor/doc` and open `index.html` (a symbolic link

to `dg/html/modules.html`) with your favorite browser.

Finally, also note the documentations of https://nvidia.github.io/cccl/thrust[thrust].



We maintain tex files in every src folder for

technical documentation, which can be compiled using pdflatex with

`make doc` in the respective src folder.



== 3. Authors, Acknowledgements, Contributions



FELTOR has been developed by Matthias Wiesenberger and Markus Held. Please see the https://feltor-dev.github.io/about/[Acknowledgements] section on our homepage

for a full list of contributors and funding.

Contribution guidelines can be found in the link:CONTRIBUTING.md[CONTRIBUTING] file.



== License



This project is licensed under the MIT license - see link:LICENSE[LICENSE] for details.