Getting started with Flamegraph

Flamegraph is a utility written by Brendan Gregg that I have recently used a lot of, and I thought I would do a writeup for people who want to get their hands dirty. If you don’t know anything at all about flamegraph I recomend watching this video from LISA13 or if you just want some slides these ones are very good.

Flamegraph is visualization graph to help identify what is going on your system. Using flamegraph requires a few steps. Let’s learn by doing.

I am using CentOS 6.4, so my instructions will need to be tweaked for Debian/Ubuntu/other users.

We need to get either perf of system tap working on our test machine. Perf is easier so we will start with that.

yum install perf

Now that perf is installed you could start recoring events now, but if you did you would find that all of the symbols would be missing and your flamegraph would have a bunch if ‘Unknown’ entries in it. To address this we need to install debug symbols of the programs we want to debug.

To start off we are going to want at least the kernel and glibc debug packages, and after that what debug symbols you want depends on what you are doing. In the examples I’m doing I want to also debug qemu-kvm, so I’ll be installing those symbols.

Installing debuginfo packages for CentOS requires you to add the debuginfo repo or download the packages manually. I’m going with the add repo option. Here is what I added:

[debuginfo]
name=CentOS-$releasever - DebugInfo
baseurl=http://debuginfo.centos.org/$releasever/$basearch/
gpgcheck=0
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6
protect=1
priority=1

Just need to install the packages now:

yum install --enablerepo=debuginfo glibc-debuginfo kernel-debuginfo qemu-kvm-debuginfo

Now that I have the debug packages installed I’m ready to start perf.

perf record -a -g -F 997 -o first.perf sleep 60

-a = all cpus -g = do call-graph (backtrace) recording -F = frequency (how often to collect data), which we are setting to 997 instead of 1000 based on Brendan’s advice (watch the video). -o = output file name. By default it will record to a file called perf.record, and it will get confusing when you are doing a lot of perf recording. sleep 60 = perf can record events for any command you do, in this case we don’t care about what we are doing, we are recording events on all cpus. So here we are just saying we want to record for 60 seconds.

• NOTE: 60 seconds can be 50-100 mb or more depending on how busy your system is.

Next we want to convert the perf file (which is a binary file) into a text file so we can process it with flamegraph.

perf script -i first.perf > first.script

Now we have a text file we can work with.

Install flamegraph:

git clone https://github.com/brendangregg/FlameGraph.git

Before running flamegraph we need to process the script file into folded stacks.

cat first.script | ./stackcollapse-perf.pl > first.folded

Lastly we will run flamegraph:

cat first.folded | ./flamegraph.pl > first.svg

Now we can look at our pretty svg file!

We can do this all with pipes as well. So instead:

perf script -i first.perf | ./stackcollapse-perf.pl | ./flamegraph.pl > first.svg

A new feature I added for flamegraph is to use a consistent palette. Let’s use this in a practical scenario.

I have 2 servers both running java inside of KVM. One is working great, the other is not. On each machine I did a perf record, just like above. I saved the files to working.script and broken.script (ran perf record, and perf script on each box).

Next I transfered the script files to my workstation, and ran:

cat working.script | ./stackcollapse-perf.pl | ./flamegraph.pl --cp > working.svg

the –cp saves the randomly generated palette into a palette.map file. Then on the same workstation I ran

cat broken.script | ./stackcollapse-perf.pl | ./flamegraph.pl --cp --colors mem > broken.svg

the –cp option will use the same palette.map that was generated from the previous run of flamegraph. This time we also have –colors mem, which uses the mem palette for any NEW symbols that the previous flamegraph did not have. This will make the differences really stand out in our flamegraph. Now a side by side comparison shows the problem:

Pretty fun stuff.