Crippling change in ld memory/cpu consumption in v2.12/v2.13

[Robert D. Kennedy]

Hello,

        I reported to the binutils e-mail list on 30-Aug-2002 a question about 
memory 
consumption in 'ld', which we observe to have increased greatly in v2.12 
relative to v2.10.

        We have narrowed this problem down to the 'ld' program itself as our 
internal 
report below describes. On a Red Hat Linux 7.3 PC, the memory increased by 
about a factor of 3 between v2.11.2 and v2.12 to link the same 65 MB 
executable. The CPU consumed increased over a factor of 3, and worsened still 
more in v2.13. Our larger 80-90 MB executables also consume 3.5X more memory, 
cause swapping on 1GB RAM machines, and consume 7-8X more CPU. Exactly the 
same code base, same OS, same machine, but different ld's.

        Is there any way to recover the clearly more efficient behavior (for 
larger 
executable builds) of 'ld' v2.11.2 in later releases? Coincidentally, the 
amount of memory used by ld v2.12 and later is about the same as the sum of 
the sizes of static libraries used to build it for a variety of our executable 
sizes.

Thanks,
Rob Kennedy
Fermi National Accelerator Laboratory
Computing Division - CDF Computing and Analysis


------------------------------------------------------------------------------


        I have completed a survey of cpu/memory consumption of different
versions of ld. The results are interesting, but first some background. I am
linking a 65 MB CDF executable, a simple lister of objects in data files, on
one uni-processor 2.53 GHz P4 machine running RHL 7.3. No other packages were
altered during the tests. I am sym-linking different builds of the ld program
in an alternate bin directory, and placing this first in my path. I add
switched to the ld command line to measure time, memory, and confirm version
used. I also used two separate RPM variants of binutils (which contains ld):
the one that comes with RedHat 7.3 and the latest bleeding edge RedHat RawHide
version. The versions of other things, the flags used, and an example of the
command line used are given at the bottom of this note.

        All 'v' versions were built from standard release binutils tarballs
from the GNU repositories. All non-'v' versions are from RPMs binutils as
packaged by RedHat. Two trials were performed for each data point to confirm
that nothing interfered with a fair comparison. All values are approximate.

ld version      time [sec]      memory [MB]
----------      ----------      -----------
v2.10.1         14.2            150
v2.11           14.2            155
v2.11.2         14.2            155
2.11.93 RHL 7.3 54              410
v2.12           52              410
v2.12.1         52              410
v2.13           64              430
2.13.90 RawHide 64              430

        My conclusion is that something changed in the source tree for ld
between v2.11.2 and v2.12 that causes this poor performance. RedHat apparently
incorporated some changes from the v2.12 into their build (not uncommon) which
also causes this behavior in RHL 7.3. Unlike my initial impression, the
performance became even WORSE in v2.13 though memory consumption worsened only
a bit.

        I looked at the ChangeLogs between these two versions (about 2000
lines), but did not find anything obvious... lots of vague possibilities.

Thanks and Good Luck,
Rob K.

- -----------------------------------------------------------------------------
--

RPM for kernel: kernel-2.4.18-5
RPM for libc6:  glibc-2.2.5-39
RPM for GNU C:  gcc-2.96-112

override LDFLAGS += --backend -Wl,--stats
override LDFLAGS += --backend -Wl,-v

/home/kennedy/Unix/bin/ld -m elf_i386 -dynamic-linker /lib/ld-linux.so.2 -o
/data0/kennedy/cdf_tests/edm_test/bin/Linux2-KCC_4_0/Edm_ObjectLister
/usr/lib/gcc-lib/i386-redhat-linux/2.96/../../../crt1.o
/usr/lib/gcc-lib/i386-redhat-linux/2.96/../../../crti.o
/usr/lib/gcc-lib/i386-redhat-linux/2.96/crtbegin.o
- -L/cdf/cdfsoft/products/msql/v2_0_10/Linux+2/lib
- -L/cdf/cdfsoft/products/msql/v2_0_10/Linux+2/lib
- -L/cdf/cdfsoft/products/msql/v2_0_10/Linux+2/lib
- -L/cdf/cdfsoft/products/msql/v2_0_10/Linux+2/lib
- -L/cdf/cdfsoft/products/oracle_client/v8_1_7_lite/Linux+2/lib/
- -L/cdf/cdfsoft/products/oracle_client/v8_1_7_lite/Linux+2/network/lib
- -L/cdf/cdfsoft/products/oracle_client/v8_1_7_lite/Linux+2rdbms/lib
- -L/opt/SUNWcluster/lib -L/cdf/cdfsoft/products/cern/2000/Linux+2.2/lib
- -L/cdf/cdfsoft/products/tcl/v8_3_1a/Linux+2.2/lib
- -L/cdf/products/prd/root/v3_03_07dKCC_4_0/Linux+2.4/lib
- -L/cdf/products/prd/dcap/f0820/Linux+2.4/lib -L.
- -L/data0/kennedy/cdf_tests/edm_test/lib/Linux2-KCC_4_0
- -L/cdf/cdfsoft/dist/releases/development/lib/Linux2-KCC_4_0
- -L/cdf/cdfsoft/products/kai/v4_0f/Linux+2.4/KCC_BASE/lib
- -L/usr/lib/gcc-lib/i386-redhat-linux/2.96
- -L/usr/lib/gcc-lib/i386-redhat-linux/2.96/../../.. --no-demangle
/data0/kennedy/cdf_tests/edm_test/tmp/Linux2-KCC_4_0/EdmModules/Edm_ObjectListe
r.o
- --stats -v --export-dynamic -lEdmModules -lFrameMods_dump -lVertexObjects
- -lJetProbObjects -lSimulationExample -lFramework -lFrameUtil -lSimulationBase
- -lHeaderObjects -lGlobalSummaryObjects -lCosmicObjects -lLevel3Objects
- -lRegionalObjects -lTopObjects -lSoftElectronObjects -lTauObjects 
-lMuonObjects
- -lElectronObjects -lJetObjects -lMetObjects -lTriggerObjects -lCalorObjects
- -lClcObjects -lSvxDaqObjects -lTofObjects -lTrackingObjects -lPADSObjects
- -lOldTofObjects -lSimulationObjects -lCalibBanks -lRawDataBanks -lRun1Banks
- -lStorableBanks -lFitting -llinalg -lRegionalGeometry -lSiliconGeometry
- -lCotGeometry -lMuonGeometry -lCalorGeometry -lClcGeometry -lTofGeometry
- -lPassiveGeometry -lStripChamberGeometry -lTrajectory -lGeometryBase -lRRL3
- -lDBViews -lAlignmentDB -lCOTCalibDBTables -lCTCCalibDBTables
- -lCalorCalibDBTables -lMuonCalibDBTables -lOffCalibDBTables
- -lShMaxCalibDBTables -lSvxCalibDBTables -lDBViews
/cdf/cdfsoft/dist/releases/development/lib/Linux2-KCC_4_0/CalibDB_RegOTL.o
- -lCalibDB_otl
/cdf/cdfsoft/dist/releases/development/lib/Linux2-KCC_4_0/CalibDB_RegText.o
- -lCalibDB_text -lCalibDB -lDBObjects -lDBObjects_msql -lmsql -lDBObjects_otl
- -lDBObjects_text -lDBObjects -lDBManager -lmsql -lmsql -lclntsh -lwtc8 -lEdm
- -lEdmUtilities -lEdmUtilities -lTrybos -lErrorLogger_i -lAbsEnv 
-lMetaDataUtils
- -lrcp -ls_i -le_l -lExperiment -lHepTuple -lCLHEP -lZMtools -lErrLogEx
- -lExceptions -lErrorLogger -lZMutility -lpacklib -lcrypt -ldl -lnsl -ltcl 
-ldl
- -lg2c -lCore -lCint -lDCache -lTree -ldl -ldcap -lpthread -lreadline 
-lhistory
- -lcurses -lm -rpath /cdf/cdfsoft/products/kai/v4_0f/Linux+2.4/KCC_BASE/lib
- -lKCC -lgcc -lc -lgcc /usr/lib/gcc-lib/i386-redhat-linux/2.96/crtend.o
/usr/lib/gcc-lib/i386-redhat-linux/2.96/../../../crtn.o

.the end.




------- End of Forwarded Message