Re: [Libunwind-devel] 10% lost unwind traces on x86-64?

[Lassi Tuura]

Hi,

To amend my previous query, we have one problem with truncated stack tracing 
without signals: global constructors for dlopen()'ed modules. This seems to be 
because there's no unwind table emitted for __do_global_ctors_aux. Any ideas if 
GCC can be coaxed to produce one? Or if anyone knows what additional tricks GDB 
employs to walk these stacks?

Here's what we get with libunwind:

UWTRACE:
 UW#0  : 0x00002b44cd96ec66 issig=0 <_ZN11IgHookTrace10stacktraceEPPvi+54>
 UW#1  : 0x00002b44cd97181a issig=0 <add+106>
 UW#2  : 0x00002b44cd971ba1 issig=0 <domalloc+65>
 UW#3  : 0x00002b44d041198d issig=0 <_Znwm+29>
 UW#4  : 0x00002b44d03ef7a1 issig=0 <_ZNSs4_Rep9_S_createEmmRKSaIcE+33>
 UW#5  : 0x00002b44d03f004b issig=0 <_ZNSs4_Rep8_M_cloneERKSaIcEm+43>
 UW#6  : 0x00002b44d03f0cd1 issig=0 <_ZNSs7reserveEm+65>
 UW#7  : 0x00002b44d03f1156 issig=0 <_ZNSs6appendERKSs+59>
 UW#8  : 0x00002b44ce84046b issig=0 <_ZNK6Reflex4Type4NameEj+619>
 UW#9  : 0x00002b44ce815fa6 issig=0 
<_ZN6Reflex8Function13BuildTypeNameERKNS_4TypeERKSt6vectorIS1_SaIS1_EEj+38>
 UW#10 : 0x00002b44ce8161b2 issig=0 
<_ZN6Reflex8FunctionC1ERKNS_4TypeERKSt6vectorIS1_SaIS1_EERKSt9type_infoNS_4TYPEE+66>
 UW#11 : 0x00002b44ce849e38 issig=0 
<_ZN6Reflex19FunctionTypeBuilderERKNS_4TypeES2_+264>
 UW#12 : 0x00002b44d4ba2bf6 issig=0 
<_ZN12_GLOBAL__N_1116__ROOT__Math__DisplacementVector3D_ROOT__Math__CylindricalEta3D_double__ROOT__Math__DefaultCoordinateSystemTag__dictEv+240>
 UW#13 : 0x00002b44d4c19918 issig=0 <_GLOBAL__I_xr.cc+32949>
 UW#14 : 0x00002b44d4c6a1b6 issig=0 <__do_global_ctors_aux+38>
 UWEND 0

Here's GDB's view of the same stack. I've seen GDB report bogus (= on stack, 
0x7ffff...) addresses for the ?? above _init, so it's not foolproof.

(gdb) where
#0  IgHookTrace::stacktrace (addresses=0x7fff339058d0, nmax=800) at 
/afs/cern.ch/user/l/lat/dev/igprof/igprof/src/IgHookTrace.cc:256
#1  0x00002b44cd97181a in add (ptr=0x11a37a40, size=134) at 
/afs/cern.ch/user/l/lat/dev/igprof/igprof/src/IgProfMem.cc:74
#2  0x00002b44cd971ba1 in domalloc (hook=<value optimized out>, n=134) at 
/afs/cern.ch/user/l/lat/dev/igprof/igprof/src/IgProfMem.cc:257
#3  0x00002b44d041198d in operator new (sz=134) at 
../../../../libstdc++-v3/libsupc++/new_op.cc:57
#4  0x00002b44d03ef7a1 in std::string::_Rep::_S_create (__capacity=109, 
__old_capacity=140734058494904, address@hidden)
    at 
/build/3jm/BUILD/slc5_amd64_gcc434/external/gcc/4.3.4/gcc-4.3.4/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:92
#5  0x00002b44d03f004b in std::string::_Rep::_M_clone (this=0x2b44d0446080, 
address@hidden, __res=47574552140352)
    at 
/build/3jm/BUILD/slc5_amd64_gcc434/external/gcc/4.3.4/gcc-4.3.4/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:604
#6  0x00002b44d03f0cd1 in std::string::reserve (this=0x7fff339073b0, 
__res=<value optimized out>)
    at 
/build/3jm/BUILD/slc5_amd64_gcc434/external/gcc/4.3.4/gcc-4.3.4/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:485
#7  0x00002b44d03f1156 in std::string::append (this=0x7fff339073b0, 
address@hidden)
    at 
/build/3jm/BUILD/slc5_amd64_gcc434/external/gcc/4.3.4/gcc-4.3.4/obj/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:330
#8  0x00002b44ce84046b in Reflex::Type::Name () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/external/slc5_amd64_gcc434/lib/libReflex.so
#9  0x00002b44ce815fa6 in Reflex::Function::BuildTypeName () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/external/slc5_amd64_gcc434/lib/libReflex.so
#10 0x00002b44ce8161b2 in Reflex::Function::Function () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/external/slc5_amd64_gcc434/lib/libReflex.so
#11 0x00002b44ce849e38 in Reflex::FunctionTypeBuilder () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/external/slc5_amd64_gcc434/lib/libReflex.so
#12 0x00002b44d4ba2bf6 in (anonymous 
namespace)::__ROOT__Math__DisplacementVector3D_ROOT__Math__CylindricalEta3D_double__ROOT__Math__DefaultCoordinateSystemTag__dict
 ()
   from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libDataFormatsMath.so
#13 0x00002b44d4c19918 in global constructors keyed to xr.cc () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libDataFormatsMath.so
#14 0x00002b44d4c6a1b6 in __do_global_ctors_aux () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libDataFormatsMath.so
#15 0x00002b44d4b7d0bb in _init () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libDataFormatsMath.so
#16 0x00002b44d35369de in ?? () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/pluginDQMDTMonitorModulePlugins.so
#17 0x0000003dee80d20b in call_init () from /lib64/ld-linux-x86-64.so.2
#18 0x0000003dee80d315 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
#19 0x0000003dee810e91 in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
#20 0x0000003dee80ce96 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#21 0x0000003dee81064c in _dl_open () from /lib64/ld-linux-x86-64.so.2
#22 0x0000003def000f9a in dlopen_doit () from /lib64/libdl.so.2
#23 0x0000003dee80ce96 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#24 0x0000003def00150d in _dlerror_run () from /lib64/libdl.so.2
#25 0x0000003def000f11 in dlopen@@GLIBC_2.2.5 () from /lib64/libdl.so.2
#26 0x00002b44cdf3c250 in edmplugin::SharedLibrary::SharedLibrary ()
   from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libFWCorePluginManager.so
#27 0x00002b44cdf35bd7 in edmplugin::PluginManager::load () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libFWCorePluginManager.so
#28 0x00002b44cdf32b68 in edmplugin::PluginFactoryBase::findPMaker ()
   from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libFWCorePluginManager.so
#29 0x00002b44cdb6c4d2 in 
edm::serviceregistry::ServicesManager::fillListOfMakers ()
   from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libFWCoreServiceRegistry.so
#30 0x00002b44cdb6dec9 in 
edm::serviceregistry::ServicesManager::ServicesManager ()
   from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libFWCoreServiceRegistry.so
#31 0x00002b44cdb6b700 in edm::ServiceRegistry::createSet () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libFWCoreServiceRegistry.so
#32 0x00002b44cda59dc8 in edm::EventProcessor::init () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#33 0x00002b44cda5f3ae in edm::EventProcessor::EventProcessor () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libFWCoreFramework.so
#34 0x000000000040d093 in main ()

(gdb) up 15
#15 0x00002b44d4b7d0bb in _init () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libDataFormatsMath.so
(gdb) i frame
Stack level 15, frame at 0x7fff33907bc8:
 rip = 0x2b44d4b7d0bb in _init; saved rip 0x2b44d35369de
 called by frame at 0x7fff33907bd0, caller of frame at 0x7fff33907bc0
 Arglist at 0x7fff33907bb8, args: 
 Locals at 0x7fff33907bb8, Previous frame's sp is 0x7fff33907bc8
 Saved registers:
  rip at 0x7fff33907bc0

(gdb) up
#16 0x00002b44d35369de in ?? () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/pluginDQMDTMonitorModulePlugins.so
(gdb) i frame
Stack level 16, frame at 0x7fff33907bd0:
 rip = 0x2b44d35369de; saved rip 0x3dee80d20b
 called by frame at 0x7fff33907c10, caller of frame at 0x7fff33907bc8
 Arglist at 0x7fff33907bc0, args: 
 Locals at 0x7fff33907bc0, Previous frame's sp is 0x7fff33907bd0
 Saved registers:
  rip at 0x7fff33907bc8

(gdb) down 2
#14 0x00002b44d4c6a1b6 in __do_global_ctors_aux () from 
/afs/cern.ch/cms/sw/slc5_amd64_gcc434/cms/cmssw/CMSSW_3_5_0/lib/slc5_amd64_gcc434/libDataFormatsMath.so
(gdb) i frame
Stack level 14, frame at 0x7fff33907bc0:
 rip = 0x2b44d4c6a1b6 in __do_global_ctors_aux; saved rip 0x2b44d4b7d0bb
 called by frame at 0x7fff33907bc8, caller of frame at 0x7fff33907ba0
 Arglist at 0x7fff33907bb0, args: 
 Locals at 0x7fff33907bb0, Previous frame's sp is 0x7fff33907bc0
 Saved registers:
  rbp at 0x7fff33907bb0, rip at 0x7fff33907bb8



Regards,
Lassi