Re: Partial linking with _RELOCATABLES - Proposed enhancement

[Ralf Wildenhues]

Hi Marc, Alexandre,

* Alexandre Duret-Lutz wrote on Sat, Mar 25, 2006 at 09:33:15PM CET:
> (I haven't read the patch yet.)

I have several half-written answers to this, but am not finished
researching yet, unfortunately.

> >>> "MA" == Marc Alff <address@hidden> writes:

>  MA> A typical "partial link" or "relocatable (-r)"  command is :
>  MA> ld -r -o glued.o part1.o part2.o ... partn.o
> 
> I didn't know "ld -r".  How portable is it?

Well, I was previously certain that there were some older systems
(without shared libraries) that did not support it.  Can't find them
now with a medium-intensive search (FWIW, even UNICOS 9 has it), so I
think the basic functionality of partial linking is pretty well
supported.  But:

- C++ code may need to use "ld -Ur" in some contexts (as the last
  partial link);  I haven't grasped all the details about when this is
  necessary.  Search for -Ur in `info ld Options' (several instances).

- MSVC's LINK has /INCREMENTAL, but from looking at documentation only,
  that creates a .exe/.dll together with a .ilk file by default, to
  which presumably you can add objects later.  I still need to check
  whether we can make this work.  (It may be useful to note, if you
  haven't noticed already, that support for MSVC in Libtool is pending
  and has been for a while.)

- There are some systems where partial linking is currently broken,
  e.g.  GNU binutils ld on w32:
  http://sourceforge.net/mailarchive/message.php?msg_id=14729192
  I have not investigated whether a newer version has this fixed yet.

Libtool uses `ld -r' for partial linking of PIC objects when the command
line for the linker is too long.  It assumes that to be slower than a
link in one step though: any linker we know how to give a script or a
response file, we prefer that.

So it seems for C++ at least the user may need to specify when that last
partial link is?  What I don't understand is: why do I have to partially
link a C++ program?  Why can't I just do a few partial links, but then a 
plain old regular link of the program output?

(I haven't checked whether compilers/linkers other than GCC/GNU binutils
have similar requirements, yet.)

>  MA> As a result, a build chain can be longer :
> 
>  MA> *.c, *.cpp --> *.o --> * glued.o --> * super-glued.o --> lib*.a, lib*.so 
>  MA> or binaries
> 
> Will your proposal allow the creation of *.a, *.so, and
> binaries out of relocatable objects?  I'm wondering, because the
> *.o should be compiled differently if they are meant to be part
> of a shared library, and the _RELOCATABLES syntax doesn't
> indicate where the object will be used.

If you want to get libtool-for-shared-libraries into play, then the
sensible thing IMHO would be to have libtool objects named .lo which
themselves are created by two partial links: the non-PIC *.o go into
output.o, the PIC ones [._]libs/*.o go into [._]libs/output.o.

Yes, Libtool does not do this ATM.  This is a bug.  I don't mind
breaking compatibility here, as that hasn't worked well for a long time.

Marc, to make this very clear to you: At the moment, this:

  $LIBTOOL --mode=link -o glued.lo input.lo

translates to
  ld -r -o glued.lo .libs/input.o

but that is not consistent, and comes from a very old libtool age when
the PIC objects were named object.lo and not .libs/object.o!  And it
will cause subsequent links using glued.lo to fail.  What should happen
instead is this:
  ld -r -o .libs/glued.o .libs/input.o
  ld -r -o glued.o input.o
  creating libtool objects glued.lo

and then "glued.lo" will be a short script just as input.lo is
(with the obvious skipping of steps when either PIC or non-PIC
objects are disabled).

>  MA> The difference between "glued.o" and "libglued.a" is that a 
>  MA> "relocatable" object
>  MA> contains all the code in one block, with internal link dependencies 
>  MA> already resolved,
> 
> Just to make sure I understand: would I get a similar result if
> instead of gluing these objects I had compiled the concatenation
> of all their sources in a single run of the compiler?

Not necessarily.  With glued.o your actual output may be larger if you
are creating a program: the linker will pick all of glued.o or nothing,
but can't pick parts of it.

This is usually what you don't want.  Except in rare cases such as
Marc's, where that is desirable.  See at end for a question to this
(*).

> (Consider that question from the point of view of someone who is
> used to split static libraries in as much objects as possible so
> the linker picks only what it needs and not more.)

Maybe Marc would also like to work around nonlinear scaling wrt. the
number of objects in linker (please correct me here if I'm wrong).

But there may also be issues with C++ and stuff like initializers,
which I haven't fully grasped yet, see above.

Then there may be difference wrt. link time optimization.  Again, I
am essentially speculating here.

>  MA> The main concern is that, how the source code is organized into 
> directories
>  MA> with recursive makefiles is an implementation choice,
>  MA> it should **not** impact what the final deliverable looks like (maybe I 
>  MA> still want ONE *.a or *.so or binary at the end,
>  MA> not expose a collection of *.a or *.so to my users).
> 
> Of your three points, this is the only one I do not understand.
> If you use Libtool convenience libraries you'll have a
> collection of *.la in your build tree for all the subparts of
> your project, but then you'll gather all these in a big *.la
> library which you will install.  So the final deliverable is
> still one library, not a collection of libraries.
> (Example in the "Libtool Convenience Libraries" section of the
> Automake manual.)

FWIW, I agree (with not understanding).

>  MA> will work both on platforms that use .o or .obj for objects,
>  MA> and in both cases the generated makefile will contain names like :
>  MA> glued.$(OBJEXT) : ...
>  MA> super-glued.$(OBJEXT) : ...
> 
> Allowing both extensions in _RELOCATABLES (and hence in the
> associated variable names) looks confusing to me.  In any way
> the resulting objects may not be built as written, so the
> extension we write here is just a convention.  I'd say let's
> offer only syntax.

I think I agree on this point.

>  MA> The generated Makefile uses LIBTOOL to actually do the linking work,
>  MA> so it should be portable : $LIBTOOL --mode=link xxx
> 
> Where xxx is "ld -r ..."?  I'd like to hear from the Libtool
> folks about this.  Is libtool supposed to emulate "ld -r" or
> something if it doesn't work?

I think xxx is "$CC -o output.$OBJEXT $OBJECTS"

and libtool will replace that by "ld -r" or whatever necessary.
I believe Marc wrote this because I suggested to him that using
plain "ld -r" may not always be the right thing.


(*) OK, here's a question that may make these relocatables completely
redudant: Libtool already offers you (with convenience archives) the
possibility to merge a collection of objects completely into a shared
library.  If I understand correctly, then all that is missing would be a
way to merge a convenience archive completely into a program as well.
Right?

If that is the case, then we could simply add a link flag to libtool
that would just do that.  And I think it would be more efficient than
the whole relocating idea as well.  So, in terms of code, this test
case code should succeed:

: ${LIBTOOL=libtool}
: ${OBJEXT=o}
: ${NM=nm}
: ${CC=gcc}
echo 'int needed() { return 0; }' > a.c
echo 'int unneeded() { return 0; }' > b.c
$LIBTOOL --mode=compile $CC -c a.c
$LIBTOOL --mode=compile $CC -c b.c
$LIBTOOL --mode=link $CC -o libconv.la a.lo b.lo
cat >main.c <<\EOF
extern int needed();
int main()
{
  return needed();
}
EOF
$CC -c main.c
$LIBTOOL --mode=link $CC -o main main.$OBJEXT \
         -use-as-whole-archive libconv.la
the_binary=.libs/main           # this is not portable.
$NM $the_binary | grep unneeded

Cheers,
Ralf