{"id":18378025,"url":"https://github.com/barbagroup/amgxwrapper","last_synced_at":"2025-04-06T22:32:05.221Z","repository":{"id":4065628,"uuid":"45640375","full_name":"barbagroup/AmgXWrapper","owner":"barbagroup","description":"AmgXWrapper: An interface between PETSc and the NVIDIA AmgX library","archived":false,"fork":false,"pushed_at":"2022-05-30T19:41:41.000Z","size":354,"stargazers_count":38,"open_issues_count":11,"forks_count":21,"subscribers_count":10,"default_branch":"master","last_synced_at":"2024-03-26T15:40:21.733Z","etag":null,"topics":["amgx","gpu","mpi","nvidia","petsc"],"latest_commit_sha":null,"homepage":"","language":"C++","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":"mit","status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/barbagroup.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":"CONTRIBUTING.md","funding":null,"license":"LICENSE","code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null}},"created_at":"2015-11-05T21:24:28.000Z","updated_at":"2024-03-20T05:51:25.000Z","dependencies_parsed_at":"2022-08-09T08:46:34.643Z","dependency_job_id":null,"html_url":"https://github.com/barbagroup/AmgXWrapper","commit_stats":null,"previous_names":[],"tags_count":12,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barbagroup%2FAmgXWrapper","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barbagroup%2FAmgXWrapper/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barbagroup%2FAmgXWrapper/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/barbagroup%2FAmgXWrapper/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/barbagroup","download_url":"https://codeload.github.com/barbagroup/AmgXWrapper/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":223264796,"owners_count":17116208,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["amgx","gpu","mpi","nvidia","petsc"],"created_at":"2024-11-06T00:30:23.631Z","updated_at":"2024-11-06T00:31:50.671Z","avatar_url":"https://github.com/barbagroup.png","language":"C++","funding_links":[],"categories":[],"sub_categories":[],"readme":"# AmgXWrapper\n\n\u003ca style=\"border-width:0\" href=\"https://doi.org/10.21105/joss.00280\"\u003e\n    \u003cimg src=\"https://joss.theoj.org/papers/10.21105/joss.00280/status.svg\" alt=\"DOI badge\" \u003e\n\u003c/a\u003e\n\u003ca style=\"border-width:0\" href=\"https://www.doi2bib.org/bib/10.21105%2Fjoss.00280\"\u003e\n    \u003cimg src=\"https://img.shields.io/badge/Cite%20AmgXWrapper-bibtex-blue.svg\"\u003e\n\u003c/a\u003e\n\nAmgXWrapper simplifies the usage of AmgX when using AmgX together with PETSc.\nA unique feature is that when the number of MPI processes is greater than the \nnumber of GPU devices, this wrapper will do the system consolidation/data \nscattering/data gathering automatically. So there's always only one MPI process\nusing each GPU and no resource competition. Though we currently only support PETSc,\nwe hope this wrapper can work with other libraries in the future.\n\nThe usage is simple. \nJust follow the procedure: ***initialize -\u003e set matrix A -\u003e solve -\u003e finalize***. \n\nFor example,\n\n```c++\nint main(int argc, char **argv)\n{\n    // initialize matrix A, RHS, etc using PETSc\n    ...\n\n    // create an instance of the solver wrapper\n    AmgXSolver    solver;\n    // initialize the instance with communicator, executation mode, and config file\n    solver.initialize(comm, mode, file);    \n    // set matrix A. Currently it only accept PETSc AIJ matrix\n    solver.setA(A);    \n    // solve. x and rhs are PETSc vectors. unkns will be the final result in the end\n    solver.solve(unks, rhs);    \n    // get number of iterations\n    int         iters;\n    solver.getIters(iters);    \n    // get residual at the last iteration\n    double      res;\n    solver.getResidual(iters, res);    \n    // finalization\n    solver.finalize();\n \n    // other codes\n    ....\n\n    return 0;\n}\n```\n\nNote, this wrapper was originally designed for our CFD solver -- \n**[PetIBM](https://github.com/barbagroup/PetIBM)**, so it may lack some features \nfor other kinds of PETSc-based applications. We are trying to make it more general.\n\n## Document\n\n1. [Dependencies](doc/dependencies.md)\n2. [Build and install](doc/install.md)\n3. [Usage](doc/usage.md)\n4. [Test](doc/test.md)\n5. [Examples](example/README.md)\n\n## Feature: system consolidation when the number of MPI processes is greater than number of GPUs\n\nIf a user ever tries to use AmgX directly without this wrapper, and when \nmultiple MPI processes are sharing a GPU device, AmgX does not deliver \na satisfying performance. One feature of this wrapper is to resolve this problem.\n\nOur solution is always to allow only one MPI process to call AmgX on each GPU.\nIn other words, only certain MPI processes can talk to GPUs when the number\nof MPI processes is greater than the number of GPUs. The wrapper will consolidate\nmatrix. Data on other processes will be gathered to the processes that can talk\nto AmgX before solving, and then scattered back afterward.\n\nFor example, if there are 18 MPI processes and 6 GPUs (12 MPI processes \nand 2 GPUs on node 1, and 6 MPI processes and 4 GPUs on node 2), only the rank 0, 6, \n12, 14, 16, 17 can talk to GPUs and will call AmgX solvers. Data on rank 1-5 \nwill be gathered to rank 0; data on rank 7-11 will go to rank 6; data on rank \n13 and 15 go to rank 12 and 14, respectively; and both rank 16 and 17 enjoy a \nsingle GPU by itself. This causes some penalties on computing time because of \ncommunications between processes. However, the overall performance is much \nbetter than calling AmgX solvers from all processes directly.\n\nThe redistribution method may be naive, but it's working. We are satisfied with\nthe performance so far.\n\n## Limitation\n\n* CPU version of AmgX solver (i.e., `hDDI` mode in AmgX or `AmgX_CPU` option\n  in our Poisson example) is not supported in this wrapper. Using PETSc's (or\n  third-party's, like Hypre) solvers and preconditioners may be a better choice\n  for CPU capability.\n* Due to our PETSc applications only use AIJ format for sparse matrices, \n  AmgXWrapper currently only supports AIJ format.\n\n## Future work\n\n* Add support of updating entry values of a matrix that is already on CUDA devices.\n* Support other matrix formats other than AIJ.\n* Add mechanisms for debugging and error handling.\n\n## Contributing\n\nTo contribute to AmgXWrapper, see [CONTRIBUTING](CONTRIBUTING.md).\n\n## How to cite\n\nTo cite this code, please use the citation or the BibTeX entry below.\n\n\u003e Pi-Yueh Chuang, \u0026 Lorena A. Barba (2017). AmgXWrapper: An interface between PETSc and the NVIDIA AmgX library. _J. Open Source Software_, **2**(16):280, [doi:10.21105/joss.00280](http://dx.doi.org/10.21105/joss.00280)\n\n\n```console\n@article{Chuang2017,\n  doi = {10.21105/joss.00280},\n  url = {https://doi.org/10.21105/joss.00280},\n  year  = {2017},\n  month = {aug},\n  publisher = {The Open Journal},\n  volume = {2},\n  number = {16},\n  author = {Pi-Yueh Chuang and Lorena A. Barba},\n  title = {{AmgXWrapper}: An interface between {PETSc} and the {NVIDIA} {AmgX} library},\n  journal = {The Journal of Open Source Software}\n}\n```\n\n## Reference\n\n[1] Chuang, Pi-Yueh; Barba, Lorena A. (2017): Using AmgX to Accelerate PETSc-Based\nCFD Codes. figshare.\n[https://doi.org/10.6084/m9.figshare.5018774.v1](https://doi.org/10.6084/m9.figshare.5018774.v1)\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarbagroup%2Famgxwrapper","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fbarbagroup%2Famgxwrapper","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fbarbagroup%2Famgxwrapper/lists"}