Data

Tools

Indexes

Applications

Experiments

Awesome

An open API service indexing awesome lists of open source software.

https://github.com/dkogan/memory_leak_instrumentation


https://github.com/dkogan/memory_leak_instrumentation

Last synced: about 1 year ago
JSON representation

Awesome Lists containing this project

README 

          
* Overview



This is a set of tools used to find memory leaks in long-running applications.

These tools use [[http://perf.wiki.kernel.org/][=perf=]] to instrument all memory allocations/deallocations. These

traces can then be analyzed to find allocated memory that was not properly

cleaned up.



This was written to find memory leaks in emacs daemon sessions. The resulting

bugs and mailing list posts:



- https://lists.gnu.org/archive/html/emacs-devel/2015-09/msg00814.html

- http://debbugs.gnu.org/cgi/bugreport.cgi?bug=21509

- http://debbugs.gnu.org/cgi/bugreport.cgi?bug=21556

- http://debbugs.gnu.org/cgi/bugreport.cgi?bug=21623



Some of the tools are emacs-specific, but some are not. Note that this is all

fairly rough, and the user would want to understand how each tool works to be

able to use it effectively. This is also not well-documented yet, but the links

above, and the text below show examples.



* Tools



** Non-emacs-specific



*** =reademacsvar.sh=

Exports some shell variables that all the scripts use to do their thing. Mostly

this is paths, etc.



*** =plotmem.sh=

Used to generate a realtime plot of memory usage of a particular process. This

lets us see the leaks as they happen.



*** =probes.sh=

Creates =perf= probes that we care about. This is all allocation/deallocations,

and anything else we like.



*** =record_alloc.sh=

Runs =perf record= to record all allocations/deallocations.



*** =parse_script.pl=

Reads the output of =perf script=, following all allocations. The output is a

list of all unexpected memory operations (calling =free()= on a pointer that

didn't come from =malloc()= for instance) and a list of all potentially leaky

memory (anything that was allocated but not freed). If we started logging some

time after the process has started, and finished logging before the process has

finished then both of these will have false positives:



- We could see a =free()= of memory that was allocated before we started logging

- We can report a leak for something that was =free()=-ed after we stopped

  logging



So take all output with a grain of salt, and use your best judgement. The

backtraces come from =perf=. It reports code addresses, and not line numbers.

You can get the line numbers line this:



#+begin_example

addr2line -e LIBRARY ADDRESS

#+end_example



where =LIBRARY= and =ADDRESS= come from the =perf= backtrace.



*** =plotleaks.sh=

Takes the output of =parse_script.pl=, and makes a plot of potential leak sizes

vs input line number. This is useful to quickly see the leaks. For instance,

let's say the emacs session we're tracing leaks 3072 bytes each time a new frame

is created, and that we created 10 frames in a row while running =perf record=.

The plot this script produces would then show 10 points at 3072 evenly spaced

through time. Leaks at the start of the session are most likely to be true ones

(there was time to =free()= the memory), so I generally follow up anything that

leaked lots of memory at the start.



*** =follow_alloc.pl=

Reads the output of =perf script=, and filters out all memory operations that do

not refer to a particular allocation size. This is useful to focus on particular

allocations identified by =plotleaks.sh=. So if =plotleaks.sh= shows lots of

leaks of size 3072, we use this to cut down the log to show only the leaks we

care about.



** Emacs-specific



*** =daemon.sh=

Starts up a new emacs daemon.



*** =client.sh=

Creates a new client frame.



*** =kill.sh=

Kills the emacs daemon.



*** =loopclient.sh=

Repeatedly creates/destroys a client frame.



*** =show_stderr.sh=

Shows the STDERR output of a process (emacs in this case). This is useful

because the emacs daemon redirects its STDERR to =/dev/null=, but gdb printing

commands such as =pr= and =pp= write to STDERR, and we want to see this output.



* General notes



When running =perf record=, the =perf= process can be overloaded and drop events

as a result (an error message says this). A larger buffer can help (=-m= option)

at the expense of using more RAM. It also helps to make smaller logs (fewer

things to record, =fp= backtrace generation instead of =DWARF=; see below).



=perf= can generate backtraces in two ways:

- Using the frame-pointer. This is the preferred method, but it only works for

  functions that have a frame pointer. All gcc optimization levels strip this

  out, so rebuild with =-fno-omit-frame-pointer= if possible.

- Using DWARF debug information. This doesn't require frame pointers, but needs

  debug info. Another down side is that this generates much bigger =perf= logs,

  and =perf= is more likely to drop events. Currently =perf= has a bug in that

  it's not able to read the split debug information in Debian packages, so you

  need this patch: http://lkml.iu.edu/hypermail/linux/kernel/1509.0/04006.html



If the binary application being instrumented is rebuilt, probes into that

application need to be deleted and re-added.



* Example



A usage example appears [[http://notes.secretsauce.net/notes/2015/10/05_memory-leak-debugging-tools.html][here]].



* License



Released into the public domain. Do whatever you like.