{"id":18707886,"url":"https://github.com/ceed/laghos","last_synced_at":"2025-05-16T18:10:01.942Z","repository":{"id":38185382,"uuid":"95588278","full_name":"CEED/Laghos","owner":"CEED","description":"High-order Lagrangian Hydrodynamics Miniapp","archived":false,"fork":false,"pushed_at":"2025-05-09T23:33:54.000Z","size":15569,"stargazers_count":192,"open_issues_count":7,"forks_count":65,"subscribers_count":35,"default_branch":"master","last_synced_at":"2025-05-10T00:27:50.504Z","etag":null,"topics":["ceed","finite-elements","high-order","hpc","hydrodynamics","lagrangian","miniapp","proxy-application"],"latest_commit_sha":null,"homepage":"http://ceed.exascaleproject.org/miniapps","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"bsd-2-clause","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/CEED.png","metadata":{"files":{"readme":"README.md","changelog":"CHANGELOG","contributing":null,"funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null,"zenodo":null}},"created_at":"2017-06-27T18:24:26.000Z","updated_at":"2025-05-09T23:33:57.000Z","dependencies_parsed_at":"2023-12-15T22:01:07.746Z","dependency_job_id":"4c53c51e-7ddd-462d-afd3-3054415aa371","html_url":"https://github.com/CEED/Laghos","commit_stats":{"total_commits":785,"total_committers":27,"mean_commits":"29.074074074074073","dds":"0.38980891719745225","last_synced_commit":"a6f5d58c5a46ddbb92bea1ea187ecd128c9e2559"},"previous_names":[],"tags_count":6,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CEED%2FLaghos","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CEED%2FLaghos/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CEED%2FLaghos/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/CEED%2FLaghos/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/CEED","download_url":"https://codeload.github.com/CEED/Laghos/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":254582907,"owners_count":22095518,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["ceed","finite-elements","high-order","hpc","hydrodynamics","lagrangian","miniapp","proxy-application"],"created_at":"2024-11-07T12:20:03.048Z","updated_at":"2025-05-16T18:10:01.878Z","avatar_url":"https://github.com/CEED.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"               __                __\n              / /   ____  ____  / /_  ____  _____\n             / /   / __ `/ __ `/ __ \\/ __ \\/ ___/\n            / /___/ /_/ / /_/ / / / / /_/ (__  )\n           /_____/\\__,_/\\__, /_/ /_/\\____/____/\n                       /____/\n\n        High-order Lagrangian Hydrodynamics Miniapp\n\n[![Build Status](https://travis-ci.org/CEED/Laghos.svg?branch=master)](https://travis-ci.org/CEED/Laghos)\n[![Build and Test (GH Actions)](https://github.com/CEED/Laghos/workflows/build-and-test-laghos/badge.svg?branch=master)](https://github.com/CEED/Laghos/actions?query=workflow%3Abuild-and-test-laghos)\n\n## Purpose\n\n**Laghos** (LAGrangian High-Order Solver) is a miniapp that solves the\ntime-dependent Euler equations of compressible gas dynamics in a moving\nLagrangian frame using unstructured high-order finite element spatial\ndiscretization and explicit high-order time-stepping.\n\nLaghos is based on the discretization method described in the following article:\n\n\u003e V. Dobrev, Tz. Kolev and R. Rieben \u003cbr\u003e\n\u003e [High-order curvilinear finite element methods for Lagrangian hydrodynamics](https://doi.org/10.1137/120864672) \u003cbr\u003e\n\u003e *SIAM Journal on Scientific Computing*, (34) 2012, pp. B606–B641.\n\nLaghos captures the basic structure of many compressible shock hydrocodes,\nincluding the [BLAST code](http://llnl.gov/casc/blast) at [Lawrence Livermore\nNational Laboratory](http://llnl.gov). The miniapp is built on top of a general\ndiscretization library, [MFEM](http://mfem.org), thus separating the pointwise\nphysics from finite element and meshing concerns.\n\nThe Laghos miniapp is part of the [CEED software suite](http://ceed.exascaleproject.org/software),\na collection of software benchmarks, miniapps, libraries and APIs for\nefficient exascale discretizations based on high-order finite element\nand spectral element methods. See http://github.com/ceed for more\ninformation and source code availability.\n\nThe CEED research is supported by the [Exascale Computing Project](https://exascaleproject.org/exascale-computing-project)\n(17-SC-20-SC), a collaborative effort of two U.S. Department of Energy\norganizations (Office of Science and the National Nuclear Security\nAdministration) responsible for the planning and preparation of a\n[capable exascale ecosystem](https://exascaleproject.org/what-is-exascale),\nincluding software, applications, hardware, advanced system engineering and early\ntestbed platforms, in support of the nation’s exascale computing imperative.\n\n## Characteristics\n\nThe problem that Laghos is solving is formulated as a big (block) system of\nordinary differential equations (ODEs) for the unknown (high-order) velocity,\ninternal energy and mesh nodes (position). The left-hand side of this system of\nODEs is controlled by *mass matrices* (one for velocity and one for energy),\nwhile the right-hand side is constructed from a *force matrix*.\n\nLaghos supports two options for deriving and solving the ODE system, namely the\n*full assembly* and the *partial assembly* methods. Partial assembly is the main\nalgorithm of interest for high orders. For low orders (e.g. 2nd order in 3D),\nboth algorithms are of interest.\n\nThe full assembly option relies on constructing and utilizing global mass and\nforce matrices stored in compressed sparse row (CSR) format.  In contrast, the\n[partial assembly](http://ceed.exascaleproject.org/ceed-code) option defines\nonly the local action of those matrices, which is then used to perform all\nnecessary operations. As the local action is defined by utilizing the tensor\nstructure of the finite element spaces, the amount of data storage, memory\ntransfers, and FLOPs are lower (especially for higher orders).\n\nThe Laghos implementation includes support for hardware devices, such\nas GPUs, and programming models, such as CUDA, OCCA, RAJA and OpenMP,\nbased on [MFEM](http://mfem.org), version 4.1 or later. These device\nbackends are selectable at runtime, see the `-d/--device` command-line\noption.\n\nOther computational motives in Laghos include the following:\n\n- Support for unstructured meshes, in 2D and 3D, with quadrilateral and\n  hexahedral elements (triangular and tetrahedral elements can also be used, but\n  with the less efficient full assembly option). Serial and parallel mesh\n  refinement options can be set via a command-line flag.\n- Explicit time-stepping loop with a variety of time integrator options. Laghos\n  supports Runge-Kutta ODE solvers of orders 1, 2, 3, 4 and 6, as well as a\n  specialized Runge-Kutta method of order 2 that ensures exact energy\n  conservation on fully discrete level (RK2Avg).\n- Continuous and discontinuous high-order finite element discretization spaces\n  of runtime-specified order.\n- Moving (high-order) meshes.\n- Separation between the assembly and the quadrature point-based computations.\n- Point-wise definition of mesh size, time-step estimate and artificial\n  viscosity coefficient.\n- Constant-in-time velocity mass operator that is inverted iteratively on\n  each time step. This is an example of an operator that is prepared once (fully\n  or partially assembled), but is applied many times. The application cost is\n  dominant for this operator.\n- Time-dependent force matrix that is prepared every time step (fully or\n  partially assembled) and is applied just twice per \"assembly\". Both the\n  preparation and the application costs are important for this operator.\n- Domain-decomposed MPI parallelism.\n- Optional in-situ visualization with [GLVis](http:/glvis.org) and data output\n  for visualization and data analysis with [VisIt](http://visit.llnl.gov).\n\n## Code Structure\n\n- The file `laghos.cpp` contains the main driver with the time integration loop\n  starting around line 609.\n- In each time step, the ODE system of interest is constructed and solved by\n  the class `LagrangianHydroOperator`, defined around line 544 of `laghos.cpp`\n  and implemented in files `laghos_solver.hpp` and `laghos_solver.cpp`.\n- All quadrature-based computations are performed in the function\n  `LagrangianHydroOperator::UpdateQuadratureData` in `laghos_solver.cpp`.\n- Depending on the chosen option (`-pa` for partial assembly or `-fa` for full\n  assembly), the function `LagrangianHydroOperator::Mult` uses the corresponding\n  method to construct and solve the final ODE system.\n- The full assembly computations for all mass matrices are performed by the MFEM\n  library, e.g., classes `MassIntegrator` and `VectorMassIntegrator`.  Full\n  assembly of the ODE's right hand side is performed by utilizing the class\n  `ForceIntegrator` defined in `laghos_assembly.hpp`.\n- The partial assembly computations are performed by the classes\n  `ForcePAOperator` and `MassPAOperator` defined in `laghos_assembly.hpp`.\n- When partial assembly is used, the main computational kernels are the\n  `Mult*` functions of the classes `MassPAOperator` and `ForcePAOperator`\n  implemented in file `laghos_assembly.cpp`. These functions have specific\n  versions for quadrilateral and hexahedral elements.\n- The orders of the velocity and position (continuous kinematic space)\n  and the internal energy (discontinuous thermodynamic space) are given\n  by the `-ok` and `-ot` input parameters, respectively.\n\n## Building\n\nLaghos has the following external dependencies:\n\n- *hypre*, used for parallel linear algebra, we recommend version 2.11.2\u003cbr\u003e\n  https://github.com/hypre-space/hypre/releases/tag/v2.11.2 \n\n- METIS, used for parallel domain decomposition (optional), we recommend [version 4.0.3](https://github.com/mfem/tpls/blob/gh-pages/metis-4.0.3.tar.gz) \u003cbr\u003e\n  https://github.com/mfem/tpls\n\n- MFEM, used for (high-order) finite element discretization, its GitHub master branch \u003cbr\u003e\n  https://github.com/mfem/mfem\n\nTo build the miniapp, first download *hypre* and METIS from the links above\nand put everything on the same level as the `Laghos` directory:\n```sh\n~\u003e ls\nLaghos/  v2.11.2.tar.gz  metis-4.0.3.tar.gz\n```\n\nBuild *hypre*:\n```sh\n~\u003e tar -zxvf v2.11.2.tar.gz\n~\u003e cd hypre-2.11.2/src/\n~/hypre-2.11.2/src\u003e ./configure --disable-fortran\n~/hypre-2.11.2/src\u003e make -j\n~/hypre-2.11.2/src\u003e cd ../..\n~\u003e ln -s hypre-2.11.2 hypre\n```\nFor large runs (problem size above 2 billion unknowns), add the\n`--enable-bigint` option to the above `configure` line.\n\nBuild METIS:\n```sh\n~\u003e tar -zxvf metis-4.0.3.tar.gz\n~\u003e cd metis-4.0.3\n~/metis-4.0.3\u003e make\n~/metis-4.0.3\u003e cd ..\n~\u003e ln -s metis-4.0.3 metis-4.0\n```\nThis build is optional, as MFEM can be build without METIS by specifying\n`MFEM_USE_METIS = NO` below.\n\nClone and build the parallel version of MFEM:\n```sh\n~\u003e git clone https://github.com/mfem/mfem.git ./mfem\n~\u003e cd mfem/\n~/mfem\u003e git checkout master\n~/mfem\u003e make parallel -j\n~/mfem\u003e cd ..\n```\nThe above uses the `master` branch of MFEM.\nSee the [MFEM building page](http://mfem.org/building/) for additional details.\n\n(Optional) Clone and build GLVis:\n```sh\n~\u003e git clone https://github.com/GLVis/glvis.git ./glvis\n~\u003e cd glvis/\n~/glvis\u003e make\n~/glvis\u003e cd ..\n```\nThe easiest way to visualize Laghos results is to have GLVis running in a\nseparate terminal. Then the `-vis` option in Laghos will stream results directly\nto the GLVis socket.\n\nBuild Laghos\n```sh\n~\u003e cd Laghos/\n~/Laghos\u003e make -j\n```\nThis can be followed by `make test` and `make install` to check and install the\nbuild respectively. See `make help` for additional options.\n\nSee also the `make setup` target that can be used to automated the\ndownload and building of hypre, METIS and MFEM.\n\n## Running\n\n#### Sedov blast\n\nThe main problem of interest for Laghos is the Sedov blast wave (`-p 1`) with\npartial assembly option (`-pa`).\n\nSome sample runs in 2D and 3D respectively are:\n```sh\nmpirun -np 8 ./laghos -p 1 -dim 2 -rs 3 -tf 0.8 -pa\nmpirun -np 8 ./laghos -p 1 -dim 3 -rs 2 -tf 0.6 -pa -vis\n```\n\nThe latter produces the following density plot (notice the `-vis` option)\n\n[![Sedov blast image](data/sedov.png)](https://glvis.org/live/?stream=../data/laghos.saved)\n\n#### Taylor-Green and Gresho vortices\n\nLaghos includes also smooth test problems that expose all the principal\ncomputational kernels of the problem except for the artificial viscosity\nevaluation. (Viscosity can still be activated for these problems with the\n`--impose-viscosity` option.)\n\nSome sample runs in 2D and 3D respectively are:\n```sh\nmpirun -np 8 ./laghos -p 0 -dim 2 -rs 3 -tf 0.5 -pa\nmpirun -np 8 ./laghos -p 0 -dim 3 -rs 1 -tf 0.25 -pa\nmpirun -np 8 ./laghos -p 4 -m data/square_gresho.mesh -rs 3 -ok 3 -ot 2 -tf 0.62 -s 7 -vis -pa\n```\n\nThe latter produce the following velocity magnitude plots (notice the `-vis` option)\n\n\u003ctable border=\"0\"\u003e\n\u003ctd\u003e \u003cimg src=\"data/tg.png\"\u003e\n\u003ctd\u003e \u003cimg src=\"data/gresho.png\"\u003e\n\u003c/table\u003e\n\n#### Triple-point problem\n\nThis is a well known three-material problem that combines shock waves and\nvorticity, thus examining the complex computational abilities of Laghos.\n\nSome sample runs in 2D and 3D respectively are:\n```sh\nmpirun -np 8 ./laghos -p 3 -m data/rectangle01_quad.mesh -rs 2 -tf 5.0 -pa\nmpirun -np 8 ./laghos -p 3 -m data/box01_hex.mesh -rs 2 -tf 5.0 -vis -pa\n```\n\nThe latter produces the following specific internal energy plot (notice the `-vis` option)\n\n\u003cimg src=\"data/tp.png\" width=\"500\" height=\"500\"\u003e\n\n## Verification of Results\n\nTo make sure the results are correct, we tabulate reference final iterations\n(`step`), time steps (`dt`) and energies (`|e|`) for the runs listed below:\n\n1. `mpirun -np 8 ./laghos -p 0 -dim 2 -rs 3 -tf 0.75 -pa`\n2. `mpirun -np 8 ./laghos -p 0 -dim 3 -rs 1 -tf 0.75 -pa`\n3. `mpirun -np 8 ./laghos -p 1 -dim 2 -rs 3 -tf 0.8 -pa`\n4. `mpirun -np 8 ./laghos -p 1 -dim 3 -rs 2 -tf 0.6 -pa`\n5. `mpirun -np 8 ./laghos -p 2 -dim 1 -rs 5 -tf 0.2 -fa`\n6. `mpirun -np 8 ./laghos -p 3 -m data/rectangle01_quad.mesh -rs 2 -tf 3.0 -pa`\n7. `mpirun -np 8 ./laghos -p 3 -m data/box01_hex.mesh -rs 1 -tf 5.0 -pa`\n8. `mpirun -np 8 ./laghos -p 4 -m data/square_gresho.mesh -rs 3 -ok 3 -ot 2 -tf 0.62831853 -s 7 -pa`\n9. `mpirun -np 8 ./laghos -p 7 -m data/rt2D.mesh -tf 4 -rs 1 -ok 4 -ot 3 -pa`\n\n| `run` | `step` | `dt` | `e` |\n| ----- | ------ | ---- | --- |\n|  1. |  339 | 0.000702 | 4.9695537349e+01 |\n|  2. | 1041 | 0.000121 | 3.3909635545e+03 |\n|  3. | 1154 | 0.001655 | 4.6303396053e+01 |\n|  4. |  560 | 0.002449 | 1.3408616722e+02 |\n|  5. |  413 | 0.000470 | 3.2012077410e+01 |\n|  6. | 2872 | 0.000064 | 5.6547039096e+01 |\n|  7. |  858 | 0.000474 | 5.6691500623e+01 |\n|  8. |  776 | 0.000045 | 4.0982431726e+02 |\n|  9. | 2462 | 0.000050 | 1.1792848680e+02 |\n\nSimilar GPU runs using the MFEM CUDA *device* can be run as follows:\n\n1. `./laghos -p 0 -dim 2 -rs 3 -tf 0.75 -pa -d cuda`\n2. `./laghos -p 0 -dim 3 -rs 1 -tf 0.75 -pa -d cuda`\n3. `./laghos -p 1 -dim 2 -rs 3 -tf 0.80 -pa -d cuda`\n4. `./laghos -p 1 -dim 3 -rs 2 -tf 0.60 -pa -d cuda`\n5. -- this is a 1D test that is not supported on the device --\n6. `./laghos -p 3 -m data/rectangle01_quad.mesh -rs 2 -tf 3.0 -pa -d cuda`\n7. `./laghos -p 3 -m data/box01_hex.mesh -rs 1 -tf 5.0 -pa -cgt 1e-12 -d cuda`\n8. `./laghos -p 4 -m data/square_gresho.mesh -rs 3 -ok 3 -ot 2 -tf 0.62831853 -s 7 -pa -d cuda`\n9. `./laghos -p 7 -m data/rt2D.mesh -tf 4 -rs 1 -ok 4 -ot 3 -pa -d cuda`\n\nAn implementation is considered valid if the final energy values are all within\nround-off distance from the above reference values.\n\n## Performance Timing and FOM\n\nEach time step in Laghos contains 3 major distinct computations:\n\n1. The inversion of the global kinematic mass matrix (CG H1).\n2. The force operator evaluation from degrees of freedom to quadrature points (Forces).\n3. The physics kernel in quadrature points (UpdateQuadData).\n\nBy default Laghos is instrumented to report the total execution times and rates,\nin terms of millions of degrees of freedom per second (megadofs), for each of\nthese computational phases. (The time for inversion of the local thermodynamic\nmass matrices (CG L2) is also reported, but that takes a small part of the\noverall computation.)\n\nLaghos also reports the total rate for these major kernels, which is a proposed\n**Figure of Merit (FOM)** for benchmarking purposes.  Given a computational\nallocation, the FOM should be reported for different problem sizes and finite\nelement orders.\n\nA sample run on the [Vulcan](https://computation.llnl.gov/computers/vulcan) BG/Q\nmachine at LLNL is:\n\n```\nsrun -n 294912 laghos -pa -p 1 -tf 0.6 -pt 911 -m data/cube_922_hex.mesh \\\n                      --ode-solver 7 --max-steps 4\n                      --cg-tol 0 --cg-max-iter 50 -ok 3 -ot 2 -rs 5 -rp 2\n```\nThis is Q3-Q2 3D computation on 294,912 MPI ranks (18,432 nodes) that produces\nrates of approximately 125419, 55588, and 12674 megadofs, and a total FOM of\nabout 2064 megadofs.\n\nTo make the above run 8 times bigger, one can either weak scale by using 8 times\nas many MPI tasks and increasing the number of serial refinements: `srun -n\n2359296 ... -rs 6 -rp 2`, or use the same number of MPI tasks but increase the\nlocal problem on each of them by doing more parallel refinements: `srun -n\n294912 ... -rs 5 -rp 3`.\n\n## Versions\n\nIn addition to the main MPI-based CPU implementation in https://github.com/CEED/Laghos,\nthe following versions of Laghos have been developed\n\n- **SERIAL** version in the [serial/](./serial/README.md) directory.\n- **AMR** version in the [amr/](./amr/README.md) directory.\n  This version supports dynamic adaptive mesh refinement.\n\n## Contact\n\nYou can reach the Laghos team by emailing laghos@llnl.gov or by leaving a\ncomment in the [issue tracker](https://github.com/CEED/Laghos/issues).\n\n## Copyright\n\nThe following copyright applies to each file in the CEED software suite,\nunless otherwise stated in the file:\n\n\u003e Copyright (c) 2017, Lawrence Livermore National Security, LLC. Produced at the\n\u003e Lawrence Livermore National Laboratory. LLNL-CODE-734707. All Rights reserved.\n\nSee files LICENSE and NOTICE for details.\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fceed%2Flaghos","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fceed%2Flaghos","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fceed%2Flaghos/lists"}