Re: [lmi] error building libxml using install_msw.sh

[Vadim Zeitlin]

On Fri, 17 Jun 2011 15:41:51 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2011-06-16 15:30Z, Vadim Zeitlin wrote:
GC> [...]
GC> >   ... during libxml2 make ...
GC> >   /bin/sh ./libtool --tag=CC --mode=link /MinGW_/bin/gcc  -g -O2 
-pedantic -W -Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch -Wco
GC> >   mment -Wtrigraphs -Wformat -Wchar-subscripts -Wuninitialized 
-Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings -Wa
GC> >   ggregate-return -Wstrict-prototypes -Wmissing-prototypes 
-Wnested-externs -Winline -Wredundant-decls  -lws2_32 -o libxml2.la -rpath
GC> >   /opt/lmi/local/lib -no-undefined -version-info 8:26:6  SAX.lo 
entities.lo encoding.lo error.lo parserInternals.lo parser.lo tree.lo
GC> >   hash.lo list.lo xmlIO.lo xmlmemory.lo uri.lo valid.lo xlink.lo 
HTMLparser.lo HTMLtree.lo debugXML.lo xpath.lo xpointer.lo xinclude.l
GC> >   o nanohttp.lo nanoftp.lo DOCBparser.lo catalog.lo globals.lo threads.lo 
c14n.lo xmlstring.lo xmlregexp.lo xmlschemas.lo xmlschemasty
GC> >   pes.lo xmlunicode.lo xmlreader.lo relaxng.lo dict.lo SAX2.lo 
xmlwriter.lo legacy.lo chvalid.lo pattern.lo xmlsave.lo xmlmodule.lo sc
GC> >   hematron.lo   -lws2_32
GC> 
GC> My most recent (successful) log has a line that's probably identical:
GC> 
GC> /bin/sh ./libtool --tag=CC --mode=link /MinGW_/bin/gcc  -g -O2 -pedantic -W 
-Wformat -Wunused -Wimplicit -Wreturn-type -Wswitch -Wcomment -Wtrigraphs 
-Wformat -Wchar-subscripts -Wuninitialized
GC> -Wparentheses -Wshadow -Wpointer-arith -Wcast-align -Wwrite-strings 
-Waggregate-return -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs 
-Winline -Wredundant-decls  -lws2_32 -o libxml2.la
GC> -rpath /opt/lmi/local/lib -no-undefined -version-info 8:26:6  SAX.lo 
entities.lo encoding.lo error.lo parserInternals.lo parser.lo tree.lo hash.lo 
list.lo xmlIO.lo xmlmemory.lo uri.lo valid.lo
GC> xlink.lo HTMLparser.lo HTMLtree.lo debugXML.lo xpath.lo xpointer.lo 
xinclude.lo nanohttp.lo nanoftp.lo DOCBparser.lo catalog.lo globals.lo 
threads.lo c14n.lo xmlstring.lo xmlregexp.lo xmlschemas.lo
GC> xmlschemastypes.lo xmlunicode.lo xmlreader.lo relaxng.lo dict.lo SAX2.lo 
xmlwriter.lo legacy.lo chvalid.lo pattern.lo xmlsave.lo xmlmodule.lo 
schematron.lo   -lws2_32

 Yes, unsurprisingly it seems to be the same.

GC> but it succeeds, and is followed immediately by:
GC>   /MinGW_/bin/gcc -shared  [...] .libs/schematron.o  -lws2_32  -o 
.libs/libxml2-2.dll [...]
GC>   Creating library file: .libs/libxml2.dll.a
GC> whereas you get the dreaded...
GC> 
GC> >   *** Warning: linker path does not have real file for library -lws2_32.
GC> >   *** I have the capability to make that library automatically link in 
when
GC> >   *** you link to this library.  But I can only do this if you have a
GC> >   *** shared version of the library, which you do not appear to have
GC> >   *** because I did check the linker path looking for a file starting
GC> >   *** with libws2_32 but no candidates were found. (...for file magic 
test)
GC> 
GC> You must have a ws2_32 DLL somewhere, right?

 Yes, but libtool looks for the import library and not the DLL. The libtool
error message says "but no candidates were found" while if I did have this
library somewhere it would have said "candidate file-name fails file magic
test" (approximate quote).

GC>   $ls /cygdrive/c/WINDOWS/system32/ws2_32*
GC>   /cygdrive/c/WINDOWS/system32/ws2_32.dll
GC> Isn't that the "shared version of the library" that libtool wants?

 No, it wants the import library. The DLL might actually pass its test as
it's pretty trivial (it looks for "pei-i386" in "objdump -f" output) but it
looks for libws2_32.*, not ws2_32.dll, so it doesn't even consider it.

GC> My (msw-xp) $PATH contains "C:\WINDOWS\system32"; could it be that
GC> you're using a 64-bit OS that doesn't expose any '*_32.dll'? I think
GC> I've read, on the Cygwin mailing list, that 32-bit libraries may be
GC> in a ghostly subdirectory that isn't really there--I can try to find
GC> a reference if it would help, but maybe you already know what I'm
GC> trying to say.

 Yes, I do, there is some file system virtualizing going on 64 bit Windows
systems and you're right, I do have 64 bit Windows version. But the purpose
of virtualizing system32 directory is exactly to provide the DLLs in
familiar places for 32 bit applications (while they really are in
c:\Windows\SysWOW64) so I don't see how could this possibly be a problem.
And in fact I do have the same file in the same place:

        $ objdump -f /cygdrive/c/Windows/System32/ws2_32.dll

        /cygdrive/c/Windows/System32/ws2_32.dll:     file format pei-i386
        architecture: i386, flags 0x0000012f:
        HAS_RELOC, EXEC_P, HAS_LINENO, HAS_DEBUG, HAS_LOCALS, D_PAGED
        start address 0x41ac145d

but libtool never checks it because it has the wrong name even though it
does occur in the path it checks which, just for reference, contains the
following directories (from libtool debug output):

        C:/opt/lmi/MinGW-20090203/bin/../lib/gcc/mingw32/3.4.5/
        C:/opt/lmi/MinGW-20090203/bin/../lib/gcc/
        /mingw/lib/gcc/mingw32/3.4.5/
        /usr/lib/gcc/mingw32/3.4.5/
        
C:/opt/lmi/MinGW-20090203/bin/../lib/gcc/mingw32/3.4.5/../../../../mingw32/lib/mingw32/3.4.5/
        
C:/opt/lmi/MinGW-20090203/bin/../lib/gcc/mingw32/3.4.5/../../../../mingw32/lib/
        /mingw/mingw32/lib/mingw32/3.4.5/
        /mingw/mingw32/lib/
        /mingw/lib/mingw32/3.4.5/
        /mingw/lib/
        
C:/opt/lmi/MinGW-20090203/bin/../lib/gcc/mingw32/3.4.5/../../../mingw32/3.4.5/
        C:/opt/lmi/MinGW-20090203/bin/../lib/gcc/mingw32/3.4.5/../../../
        /mingw/lib/mingw32/3.4.5/
        /mingw/lib/
        /lib/mingw32/3.4.5/
        /lib/usr/lib/mingw32/3.4.5/
        /usr/lib/
        /usr/local/bin
        /usr/bin
        /cygdrive/c/Windows/system32
        /cygdrive/c/Windows
        /cygdrive/c/Windows/System32/Wbem
        /cygdrive/c/Windows/System32/WindowsPowerShell/v1.0

I don't know where does half of them (e.g. all "/mingw/*" ones?) come from
but, as you can see, C:/opt/lmi/MinGW-20090203/lib doesn't appear in it and
hence the only version of libws2_32.lib on the system can't be found.

GC> >  But the real problem is, of course, that it can't find ws2_32 library in
GC> > the first place.

 So, to summarize, I still believe this is the case. I'd love to know
where and how does it find it on your system. If you could temporarily
modify your libtool to show the libraries it finds, it would be great. A
simple way to do it is to search for "but no candidates are found" in
/opt/lmi/xml-scratch/libxml2-2.6.26/libtool script and then insert

        echo "!!! $potlib"

before "break 2" line 12 lines above it (I guess I could make an ed script
to do this but it's probably faster to do it interactively). After this you
can run the command

        ./libtool --tag=CC --mode=link /MinGW_/bin/gcc  -g -o libxml2.la \
                -rpath /opt/lmi/local/lib -no-undefined -version-info 8:26:6 \
                *.lo -lws2_32

interactively in /opt/lmi/xml-scratch/libxml2-2.6.26 directory. What does
it output for you?


GC> Are you using exactly the same nearly-minimal Cygwin installation as we are,
GC> as produced by 'install_cygwin.bat'?

 Yes, absolutely. I finally succumbed to installing vim as I couldn't stand
the idea of using notepad one more time but other than that it's exactly as
installed by the script.

GC> I'm just wondering whether you have your own Cygwin with extra
GC> components that take precedence over the MinGW stuff.

 I did try installing w32api package which installed Cygwin libws2_32.a
library but this failed to placate libtool and this is how I know that it
gives a different error when the library is present but magic test fails
(this and half an hour spent on reading its code...) as it still failed
because MinGW objtool didn't find Cygwin library anyhow. But I uninstalled
this package later and did _not_ install mingw-w32api one as I realized
that I already had its contents under /MinGW_ mount and had just to use the
correct -L option.

GC> OTOH, that may not matter, because our libtool comes from Cygwin:
GC>   $ls -l /bin/libtool
GC>   -rwxr-xr-x 1 Arktos root 322116 2010-09-23 19:48 /bin/libtool

 Please be careful with this because this libtool is *not* the one that
is used for libxml build. That libtool is generated during libxml configure
from the files included in its distribution and is different from the
system one which would only be used if we ran libtoolize on libxml
ourselves before configuring it. In particular Cygwin 1.7 libtool is of
recent (2010) 2.4 variety while the one included in libxml is an ancient
1.5.22 (from 2005).

 But in any case both of them are, of course, Cygwin shell scripts for the
simple reason that MSYS is not even installed here.


GC> It's intended to be usable even by people who don't know what a shell script
GC> is for. If it fails, then all they can do is send me their log; and early
GC> exits won't really make it easier for me to diagnose their problems.

 How do they even know that it failed though? It doesn't even output an
error in this case.

GC> However, it's most often used by Wendy or me. It takes about an hour to run
GC> on my dedicated test machine, and it would be nice to have it stop after
GC> fifteen minutes if it's certain to fail. But that's not a high priority,
GC> and because of its intended use we spend a lot of time testing every change
GC> rigorously, so I might not take the time to change it soon. Couldn't you
GC> get the behavior you prefer if you manually specify 'set -e'?

 Yes, sure, I could do this. I just thought that finishing with an
optimistic "Finished building LMI" message when the build had totally
failed was dangerously misleading...


 Anyhow, what should we do about the ws2_32.dll problem? I could install 32
bit Win7 system and repeat all of it there which would allow to at least
detect whether the problem is 64-bit-specific. Or we could just add
-L/opt/lmi/MinGW-.../lib option as I suggested because it definitely
shouldn't hurt anything.

 In the meanwhile I can't help but notice that this is yet another problem
that would have been much simpler to debug and actually probably wouldn't
have occurred at all if we didn't use this unstable mix of MinGW and
Cygwin. The fact that many paths are not understood by a half of tools used
surely makes the life more interesting, especially considering that it's a
different half every time... I'm just ranting, of course, as I realize that
there is no chance of any big changes to the build system in the near
future but anything at all, either MinGW/MSYS or Cygwin on its own or even
MSBuild with cmd.exe, would be better than the current setup IMO.

 Regards,
VZ