Re: Speeding up libtool

[Ralf Wildenhues]

Hi Robert,

* Robert Ögren wrote on Mon, Mar 14, 2005 at 11:11:00PM CET:
> 
> I looked through the TODO thread from 2004-11 and saw some talk about 
> speeding up libtool. I'm particularly interested in improved compilation 
> speed on Cygwin (Windows). Is there any progress on this?

For the HEAD branch of Libtool, I have made some changes that cause it
to fork less often.  For the particular bad case I was looking at
(linking against ~500 objects), I got it from about 11 sec to about 5
sec, both of which include 2.5 sec of time spent actually linking (all
data cached) on a fast linux/x86_64 machine.

The improvements also benefit the general case, but are most likely less
pronounced, even if Cygwin should be the candidate to benefit most.

If you have resources/time to test, it would be nice to see a comparison
for your projects.  For best results on Cygwin, you should probably
configure (with libtool HEAD) like
  CONFIG_SHELL=/bin/ash lt_ECHO='printf %s' /bin/ash configure [...]

(replace /bin/ash with the path to Cygwin's Almquist shell clone).
I have wanted to get this to work right automatically, but have not had
time to look into it (the shell/ECHO selection is mixed Libtool/Autoconf
code and focuses on portability more than speed -- there have been old
and quite buggy ash variants on some systems).

The speedup between using bash and using ash should be quite noticeable
on Cygwin!

> A while ago I created a temporary solution for myself which caches the 
> commands executed by libtool 1.5 to speed up the compilation process. It 
> is a small Perl script that is run in place of libtool and that 
> maintains a simple database of libtool command line => resulting 
> commands by running the real libtool any time a command is not already 
> in the database and parsing its standard out and storing the commands 
> and .la files. It understands very little of what libtool actually does, 
> the speedup is achieved by invoking libtool less often.

This is a very nice idea!  The hairy stuff is in the details, however.

> By replacing the name of the source file and the corresponding output 
> file with placeholders if they match it usually only requires a few 
> libtool invocations for compiling many source files. The link commands 
> are usually unique but at least the second time a source tree is rebuilt 
> they are already cached.

Ah, ok.  For use within the same $buildtree and compilation only this
should be mostly fine (save locking, as mentioned in your README).

Linking is a problem, though: shell wrappers break, as do generating
symbol lists.  Can't think of more right away, but I'm sure there are
many more corner cases which break.

> This is aimed at software developers like myself who build projects from 
> CVS and frequently modify some source files, compile and relink.
> 
> The code was not written with portability in mind and certainly has bugs 
> and quirks, but works well for me and should be possible to port to 
> other platforms than Cygwin. The speedup might not be very large on 
> those platforms however. But if anyone tries, please tell me how it worked.

How much is the actual speedup in numbers?  How much with the HEAD
branch of Libtool?  Iff this gives a decent speedup, I think the idea
could belong in Libtool proper.  That is, if you and the other Libtool
developers are so inclined.

Thinking out loud more: if this were created as output of `configure'
(possibly acting upon some configure and/or libtoolize switch), it would
also be possible to make libtool-cache (more) portable and much more
foolproof.  For example, if configure decides locking is necessary, the
cache could be bypassed.  $objdir, PATH_SEPARATOR, etc. would be
trivially available, the `libtoolcacheize' beast could be killed..

We could add a marker whenever libtool does non-cacheable things so that
the cache would be bypassed then.  Or just let libtool output
"everything" it does so that even writing wrappers would work.
Maybe into a separate file descriptor.

Most likely the cache would have to be invalidated whenever
config.status changes.  And it would have to be $top_builddir-local, so
you'd have several caches for several subpackages.

But for all of this to work reasonably, I think libtool needs to be
audited so it outputs the executed commands very closely.

I'd also think it would be possible to rewrite libtool-cache in portable
shell, but then remaining speedup would have to be evaluated (one `perl'
exec is, after all, not cheap).  I'm pretty sure this would be more
useful as a separate script than being directly integrated into the
`libtool' script, since many shells need a long time for the initial
parsing of `libtool'.

Minor nit for your current script: Why do you remove comments from the
created .la files?  I don't see a benefit here.

What do you think?  What do the other Libtool developers think?
Note that it's probably not worth investing work unless there is a
demonstrable speedup involved.

Regards,
Ralf