freenode
Languages & Toolchains

Prototype dsymutil rewrite is about 10x faster on Chromium

Nico Weber's lld-style DWARF linker cuts Chromium dSYM times from minutes to seconds and uses far less memory, and LLVM's dsymutil maintainer is open to landing it as a replacement mode.

Nico Weber has proposed a new implementation of LLVM's dsymutil that, on Chromium-scale inputs, links debug info about ten times faster than today's parallel mode and about thirty times faster than the classic linker, while using much less memory.

dsymutil turns compiler DWARF into the dSYM bundles Apple's debugger and crash symbolication expect. A parallel mode already made the tool roughly three times faster, but Weber wrote that Chromium still takes minutes, while lld's PDB linker finishes comparable debug-info linking in seconds. His prototype borrows lld's approach: sequential parallel regions, less work, more raw copying of debug data, and less deserialize-and-reserialize.

On a Chromium build with line tables only (about 39,000 objects and 3.9 GB of DWARF in), the prototype finished in 1.7 seconds with a 4.3 GB peak, against 19.1 seconds and 15.7 GB for the parallel linker and 74.4 seconds for classic. With full debug info (about 40,000 objects and 44 GB in) it took 12.2 seconds and 18 GB, versus 140 seconds and 87 GB parallel, and 333 seconds classic. A ThinLTO build was similar, at 12.3 seconds and 18 GB against 165 seconds and 105 GB. Output was also somewhat smaller.

Weber said lldb's dSYM API tests pass, llvm-dwarfdump verification is clean, symbolizer results match on 220,000 sampled addresses, and Breakpad's dump_syms output matches. The prototype is deterministic; today's parallel linker was not, for full debug info plus ThinLTO. That bug was already fixed by a change Jonas Devlieghere landed around the same time.

Devlieghere, who maintains dsymutil, said he is interested and will review incremental changes. Identical output is straightforward to land; DWARF that actually changes needs revalidation, though he has a prototype diffing tool for that. After looking at the code he said liveness, ODR uniquing, and ordering already match the existing model, and asked whether a separate tool is required or just an artifact of prototyping.

Weber replied that a separate binary is not necessary. It could be another linker mode inside dsymutil, with the intent of eventually replacing the current parallel mode. He argued a rewrite is easier than moving the existing parallel linker over: the lld-style loop structure is simpler, skipping abbreviation-table re-uniquing costs little size and avoids building DIE trees, and LLVM's Mach-O output often already has addresses in debug sections, so relocations can be skipped. The prototype does not use the existing DWARF linker library. He asked to start sending changes if that replacement approach is acceptable.

David Blaikie, separately, said a similarly fast DWARF-aware linker for dwp would be valuable at Google if it could shrink packaged debug info without a large memory increase, and hoped the linking logic could eventually be shared.