Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/llnl/mpileaks

Tool to detect and report leaked MPI objects like MPI_Requests and MPI_Datatypes
https://github.com/llnl/mpileaks

mpi shell

Last synced: 6 days ago
JSON representation

Tool to detect and report leaked MPI objects like MPI_Requests and MPI_Datatypes

Awesome Lists containing this project

README 

        

===================================================================

Introduction

===================================================================



mpileaks is a library to detect application leaks of MPI objects. Using

the MPI profiling interface, it keeps track of all MPI calls that

"allocate" and "free" objects.  Any unfreed objects are identified and

reported either during MPI_Finalize or whenever MPI_Pcontrol is called.



mpileaks displays the stack trace where the leak occurred. The trace

includes file name and line number of the allocation call. Since many

MPI tasks often have the same leak, mpileaks computes and reports

the total number of objects leaked across all processes for a given

source code location.  To acquire stack traces, mpileaks internally

uses Todd Gamblin's Callpath class.



https://github.com/scalability-llnl/callpath



To obtain stack traces, applications must be compiled with the

'-g' option. However, it is not necessary to link to mpileaks to

use it.  One may select to use mpileaks at run time with existing

executables (see using mpileaks below). 



The following objects are tracked:

   x ==> supported

   - ==> still todo



   x Requests

     x pt2pt

     x persistent pt2pt

     x nonblocking file IO

     x generalized requests

   x Datatypes

   x Groups

   x Communicators

     x dup / split / create

     x intercomm create / merge

     x topology

     x dynamic processes

   x User-defined reduction ops

   x Files

   x Windows

   x Info objects

   x Attribute keyvals

   x Errhandlers (need example)

   x Memory

   - Ports

   - Published names



===================================================================

Building and using mpileaks

===================================================================



A basic build and install:



  ./configure \

    --prefix= \

    --with-adept-utils= \

    --with-callpath=

  make

  make install



Just before running, set:



  export LD_PRELOAD="/lib/libmpileaks.so"



By default, mpileaks reports a single line from the stacktrace,

but one may increase the number of lines by setting the

$MPILEAKS_STACK_DEPTH environment variable.  For example, to

get two lines:



  export MPILEAKS_STACK_DEPTH=2



To get the entire stacktrace, set this value to -1.



Also, by default, mpileaks chops off the first 4 function calls

in the stack trace, which are internal to the mpileaks library.

One can adjust the number of functions removed from the

initial part of the trace by setting $MPILEAKS_STACK_START to

any non-negative integer (the default value is 4).  This feature

is useful for MPI implementations in which the Fortran API is

just a thin wrapper around the C API. For instance, if the

Fortran-to-C wrapper imposes one more level of function calls

between the application and the MPI library, one may increase

this number from 4 to 5 to shave off this single layer so that

the stack trace begins in application code.



As a convenience, mpileaks installs SLURM srun wrappers.

It creates an srun-mpileaks wrapper for C and C++ codes and

another srun-mpileaksf wrapper for Fortran applications.

These scripts run the application within an environment that

sets LD_PRELOAD, MPILEAKS_STACK_START, and MPILEAKS_STACK_DEPTH.

This way, the user simply must replace "srun" with the

corresponding script to enable mpileaks.



Different systems and different MPI libraries may require

different values for MPILEAKS_STACK_START.  There are configure

options to set different values in the srun wrappers.  For example,

to set MPILEAKS_STACK_START to 4 in srun-mpileaks and to 6 in

srun-mpileaksf, specify the following during configure:



  --with-stack-start-c=4

  --with-stack-start-fortran=6



===================================================================

Extending MPILEAKS to handle other types

===================================================================

   

mpileaks is really a framework to detect mismatched function calls. As

such it can be extended to detect other types of leaks. To use this

framework, one must: (a) register the handle type used to uniquely

identify the function calls of interest; and (b) use the provided

functions 'allocate' and 'free' to associate a function call as

allocating or freeing resources. We provide an example below that,

for any given program, identifies MPI open files that are not closed.  



--

#include "mpi.h"

#include "mpileaks.h"                 /* Handle2Callpath */ 



/*

 * Instantiate class with the specific handle-type that identifies

 * one function call from another (MPI_File in this case). 

 * Define 'is_handle_null' for your specific handle type. 

 */

 

static class MPI_File2Callpath : public Handle2Callpath

{

public: 

  bool is_handle_null(MPI_File handle) {

    return (handle == MPI_FILE_NULL) ? 1 : 0; 

  }

} File2Callpath; 



/* 

 * Associate function calls as "allocate" or "free" functions. 

 */ 



int MPI_File_open(MPI_Comm comm, char *filename, int amode, 

                  MPI_Info info, MPI_File *fh)

{

  int rc = PMPI_File_open(comm, filename, amode, info, fh); 

  File2Callpath.allocate(*fh); 

  return rc; 

}



int MPI_File_close(MPI_File *fh) 

{

  MPI_File handle_copy = *fh; 

  

  int rc = PMPI_File_close(fh);  

  File2Callpath.free(handle_copy); 

  

  return rc; 

}