[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Speeding up libtool
From: |
Robert Ögren |
Subject: |
Re: Speeding up libtool |
Date: |
Tue, 15 Mar 2005 20:59:39 +0100 |
User-agent: |
Mozilla Thunderbird 1.0 (Windows/20041206) |
Ralf Wildenhues wrote:
Hi Robert,
Hi Ralf,
For the HEAD branch of Libtool, I have made some changes that cause it
to fork less often. For the particular bad case I was looking at
(linking against ~500 objects), I got it from about 11 sec to about 5
sec, both of which include 2.5 sec of time spent actually linking (all
data cached) on a fast linux/x86_64 machine.
The improvements also benefit the general case, but are most likely less
pronounced, even if Cygwin should be the candidate to benefit most.
If you have resources/time to test, it would be nice to see a comparison
for your projects. For best results on Cygwin, you should probably
configure (with libtool HEAD) like
CONFIG_SHELL=/bin/ash lt_ECHO='printf %s' /bin/ash configure [...]
Interesting, I'll have to find some time to try that soon.
By replacing the name of the source file and the corresponding output
file with placeholders if they match it usually only requires a few
libtool invocations for compiling many source files. The link commands
are usually unique but at least the second time a source tree is rebuilt
they are already cached.
Ah, ok. For use within the same $buildtree and compilation only this
should be mostly fine (save locking, as mentioned in your README).
Linking is a problem, though: shell wrappers break, as do generating
symbol lists. Can't think of more right away, but I'm sure there are
many more corner cases which break.
Shell wrappers? If you mean the shell scripts that are created to set up
PATH etc. before running an uninstalled EXE from .libs, they are also
saved by libtool-cache by using the same mechanism that is used to save
the .la{,i} files (serialize_ltfile) and it seems to work.
I just open the generated files, transform them into a set of echo
statements that recreate the files, and splice them into the command
sequence that Libtool outputs. The corresponding wrapper .exe file that
is generated by Libtool when running on Cygwin is just copied from a
precompiled version.
I don't handle the generation of symbol lists apart from ignoring the
line that Libtool prints when it's generating it, so that is probably
broken. I haven't needed it yet...
How much is the actual speedup in numbers?
I will have to get back to you after doing some proper benchmarks, but
you can find at least some numbers here:
http://mail.gnome.org/archives/gtk-devel-list/2005-March/msg00062.html
This is on a quite fast machine with lots of RAM for disk cache, however
I did leave the antivirus software running (although set to ignore the
source tree). I should turn that off and see if it makes any difference.
> How much with the HEAD branch of Libtool?
Sorry, I must admit I haven't actually tried it yet, although I did
check out the source from anoncvs recently. I'll try it soon.
Iff this gives a decent speedup, I think the idea
could belong in Libtool proper. That is, if you and the other Libtool
developers are so inclined.
I certainly wouldn't mind this functionality being included in Libtool
(or perhaps just becoming unnecessary if Libtool is rewritten in C or
bytecode). If anyone wants to they should feel free to improve my crude
code or just take the useful ideas and reimplement them properly. I
can't promise that I'll have time myself to do that implementation.
Thinking out loud more: if this were created as output of `configure'
(possibly acting upon some configure and/or libtoolize switch), it would
also be possible to make libtool-cache (more) portable and much more
foolproof. For example, if configure decides locking is necessary, the
cache could be bypassed. $objdir, PATH_SEPARATOR, etc. would be
trivially available, the `libtoolcacheize' beast could be killed..
That would be great.
We could add a marker whenever libtool does non-cacheable things so that
the cache would be bypassed then. Or just let libtool output
"everything" it does so that even writing wrappers would work.
Maybe into a separate file descriptor.
Yes, that would be nice. Right now I remove some messages that Libtool
writes, and try to detect warning messages and skip caching in such
cases. But it is not perfect.
Most likely the cache would have to be invalidated whenever
config.status changes. And it would have to be $top_builddir-local, so
you'd have several caches for several subpackages.
But for all of this to work reasonably, I think libtool needs to be
audited so it outputs the executed commands very closely.
Agreed.
I'd also think it would be possible to rewrite libtool-cache in portable
shell, but then remaining speedup would have to be evaluated (one `perl'
exec is, after all, not cheap). I'm pretty sure this would be more
useful as a separate script than being directly integrated into the
`libtool' script, since many shells need a long time for the initial
parsing of `libtool'.
Also agree.
Minor nit for your current script: Why do you remove comments from the
created .la files? I don't see a benefit here.
That is just to reduce the size of the cache for speed. I don't actually
generate the .la files and wrapper script, I just save the ones that
Libtool generates, which gives a rather long command line to save, even
without comments.
I use an embarassingly simple "database" format - just a flat file where
each line contains first the Libtool command line, a separator, and then
the corresponding set of commands to be executed. Then I just read the
file sequentially until the desired command is found or EOF occurs.
(There are actually three cache files, one for compiling, one for
linking libs and one for linking exes.)
That was just the first try, and it worked well enough so I didn't
bother using a better database format. I'm sure there are Perl modules
that can help with this.
Note that it's probably not worth investing work unless there is a
demonstrable speedup involved.
I completely agree. I'll try to make some better test runs and post the
results here soon. I'll try to script the tests as far as possible to
make it easier for other people to try and reproduce the results. It
would also be useful to confirm that there indeed exist people who need
a faster Libtool, to justify the added complexity of caching. My current
code works fine for my modest needs, so don't spend time on it unless
you think it will benefit other people.
To be honest, so far I only know of one person besides me that actually
have used libtool-cache (hi Tor :). There might be others, but they
haven't told me and SourceForge's statistics system is currently broken.
I haven't really promoted the code either, didn't feel it was
sufficiently tested.
Robert
- Speeding up libtool, Robert Ögren, 2005/03/14
- Re: Speeding up libtool, Ralf Wildenhues, 2005/03/15
- Re: Speeding up libtool, Tor Lillqvist, 2005/03/15
- Re: Speeding up libtool,
Robert Ögren <=
- Re: Speeding up libtool, Max Bowsher, 2005/03/15
- Re: Speeding up libtool, Ralf Wildenhues, 2005/03/16
- Re: Speeding up libtool, Robert Ögren, 2005/03/24
- Re: Speeding up libtool, Ralf Wildenhues, 2005/03/25
- Re: Speeding up libtool, Robert Ögren, 2005/03/25
- Re: Speeding up libtool, Robert Ögren, 2005/03/25
- Testsuite failures on Cygwin with ECHO=printf (was: Re: Speeding up libtool), Robert Ögren, 2005/03/26
Re: Speeding up libtool, Robert Ögren, 2005/03/25