Analyzing xperf traces is virtually impossible if you don’t have symbols. Unfortunately the xperf toolkit makes it easy to end up with symbols that takes hours to load, or don’t load at all.
After two people contacted me the same day (one asking why symbol loading didn’t work, the other asking why it was too slow) I decided it was time to document this issue.
Update, November 2014
It’s still important to copy known-good copies of to the WPT install directory, since otherwise you can’t be sure which DLL versions you will get (if any). The VS 2013 versions seem to be good. But, for some PDBs a new problem has surfaced. It shows up when profiling the Chrome web browser. You can read about this new problem and its solution at Slow Symbol Loading in Microsoft’s Profiler, Take Two. And, this new episode shows the importance of profiling your profiler, if only to see whether time is being spent in dbghelp.dll, msdia120.dll, or somewhere else.
Update, October 2015
UIforETW ships a workaround for this problem. From the UIforETW Settings dialog you just need to click the Copy symbol DLLs button to copy known-good DLLs to the WPT install directory which should avoid this problem. UIforETW also ships with a workaround for the new problem found with Chrome’s symbols – see the link in the pargraph above.
Now we return you to your regularly scheduled blog post.
Symbol loading is handled by two DLLs – symsrv.dll (which retrieves PDB files from your symbol server) and dbghelp.dll (which scans through the PDBs). Unfortunately, the xperf toolkit no longer ships with these DLLs. That means that some people will be able to load symbols, and some people won’t – it all depends on what happens to be in your path.
With xperfview it could be very hard to diagnose symbol loading problems. WPA has a Diagnostic Console that gives you some hope of figuring out what is going on. It was the diagnostic console that first told me that my colleague couldn’t load symbols because he didn’t have symsrv.dll, and it is the diagnostic console that tells you what symbols are really being loaded – ignore the inaccurate progress meter.
Foreshadowing: newer is not better
The obvious place to get symsrv.dll and dbghelp.dll is from the latest version of windbg. After all, newer is shinier, and shinier is better, right? And doesn’t it make sense to get symsrv.dll and dbghelp.dll from the same Windows 8 SDK that you got the xperf toolkit from?
The way xperf toolkit uses PDBs is very different from how a debugger uses PDBs. Xperf (whether you are using xperfview or WPA or GPUView) scans through all of the symbols in the PDBs and stores the transcoded results in .symcache files in the directory specified by the _NT_SYMCACHE_PATH environment variable. This is great because these transcoded .symcache files load really quickly, and they are archived in a compact format so that if you trace a temporary build you can still analyze it a few days later when the PDB files are long gone. So far so good.
However, for some reason the most recent versions of dbghelp.dll do this scanning extremely slowly. One of our PDBs took 2.5 hours to load. That’s 150 minutes. I’m a patient man, but if symbols don’t load while I’m making coffee then I get annoyed.
Yes, I recorded a trace of WPA analyzing a trace. How recursive. But not very helpful. The trace (once its symbols loaded) showed me that symbol decoding is single threaded (boo) and CPU bound in dbghelp.dll, but it did not directly lead to a solution.
When I found a PDB which took 150 minutes to transcode I used it as a test case. I deleted the associated .symcache file so that it would have to be transcoded again and then I switched to using an older version of dbghelp.dll
It was faster. A lot faster. It transcoded the file in one minute. For those of you watching from home that is:
2.5 hours * 60 min/hr / 1.0 minutes = 150 hr*min/hr*min = 150
If I’ve done my math correctly then the old version of dbghelp.dll was 150 times faster, which in metric is ‘awesome’. It makes the difference between ETW being usable, and not.
DLL hell, solved
If you use UIforETW then the problem is easily solved by clicking Copy symbol DLLs from the UIforETW Settings dialog.