Re: [lmi] Strange LMI installation failure under Windows 7 SP1: /cache_for_lmi/downloads not found

[Vadim Zeitlin]

On Wed, 20 Aug 2014 14:01:52 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2014-08-18 16:30Z, Vadim Zeitlin wrote:
GC> > 
GC> >  I've set up a fresh virtual machine with Windows 7 SP1 for LMI testing.
GC> > Following the usual instructions, I've installed Cygwin without problems
GC> > and launched install_msw.sh. After just a few minutes, the script
GC> > terminated with "Finished building lmi." message but I quickly realized
GC> > that actually LMI wasn't built and was nowhere to be found under /opt/lmi.
GC> 
GC> It's been a long while since I've run that script myself, or updated
GC> Cygwin on my machine. They recently updated their coreutils binaries
GC> after a long hiatus, so I wonder if that might be the cause somehow.

 I don't think so, everything worked just fine once it was really there
and I think I have a more plausible explanation below.

GC> > Also, further down in the log there is
GC> > 
GC> >   cd /cache_for_lmi/downloads && echo "6bba3bd1bf510d152a42d0beeeefa14d 
*binutils-2.19.1-mingw32-bin.tar.gz" | md5sum --check
GC> >   /bin/sh: line 0: cd: /cache_for_lmi/downloads: No such file or directory
GC> >   install_mingw.make:251: recipe for target 
'binutils-2.19.1-mingw32-bin.tar.gz' failed
GC> >   make: *** [binutils-2.19.1-mingw32-bin.tar.gz] Error 1
GC> > 
GC> > However the next command does find the directory successfully:
GC> > 
GC> >   cd /cache_for_lmi/downloads && echo "ada423c49bc5bfa7c3e7a80a711c2a1a 
*mpatrol_1.4.8.tar.gz" | md5sum --check
GC> >   mpatrol_1.4.8.tar.gz: OK
GC> 
GC> So 'cd' fails at first, but later succeeds with the same directory.

 This is indeed what happens and I now know the reason:
install_mpatrol.make explicitly creates this directory in its initial_setup
target, which is a dependency of "all". However install_mingw.make does not
do this, and as the directory doesn't exist initially (see below for why),
it fails.

GC> Maybe 'mkdir' functions in a "lazy" way due to caching?
GC> 
GC> >  I don't understand at all what's going on here. My hypothesis is that
GC> > mounting C:/cache_for_lmi on /cache_for_lmi has somehow temporarily
GC> > confused Cygwin but this seems pretty unconvincing even to me.
GC> 
GC> Me too. Could there have been a conflicting mount point for this
GC> directory before the script was run?

 No, just for the record, it was a fresh Windows 7 SP1 installation,
without anything

GC> The script explicitly
GC> contemplates that possibility, and tries to work around it; but
GC> maybe the 'mount' command's behavior has changed. Does your
GC> /etc/fstab offer any clue? Mine says:
GC>   C:/cache_for_lmi          /cache_for_lmi lmi_specific binary,user 0 0

 Hmm, I don't have anything in /etc/fstab:

        % cat /etc/fstab
        # For a description of the file format, see the Users Guide
        # http://cygwin.com/cygwin-ug-net/using.html#mount-table

        # This is default anyway:
        none /cygdrive cygdrive binary,posix=0,user 0 0

This is suspicious as install_cygwin.bat clearly writes to this file. But
somehow -- even though I received "Cygwin installation seems to have
succeeded" message at the end -- this didn't seem to have had any effect.
Perhaps it's another change in Cygwin 1.7?

 FWIW I do have cache_for_lmi mounted on c:/cache_for_lmi:

        % mount
        C:/opt/lmi/MinGW-20090203 on /MinGW_ type ntfs (binary,user)
        C:/cygwin-lmi/bin on /usr/bin type ntfs (binary,auto)
        C:/cygwin-lmi/lib on /usr/lib type ntfs (binary,auto)
        C:/cache_for_lmi on /cache_for_lmi type ntfs (binary,user)
        C:/cygwin-lmi on / type ntfs (binary,auto)
        C:/opt/lmi on /opt/lmi type ntfs (binary,user)
        C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)

GC> Do you have a directory (or other file) such as
GC>   C:\cygwin-lmi\cache_for_lmi
GC> in the Cygwin directory? That could cause confusion.

 Yes, there is indeed such directory and, more interestingly, it has
downloads subdirectory which is empty:

        % ls -AR /cygdrive/c/cygwin-lmi/cache_for_lmi
        /cygdrive/c/cygwin-lmi/cache_for_lmi:
        downloads

        /cygdrive/c/cygwin-lmi/cache_for_lmi/downloads:


 So I finally think I see what's going on here:

0. For some unknown but not really unexpected reason, Cygwin installation
   script doesn't mount /cache_for_lmi on C:/cache_for_lmi. It is still
   unknown why /etc/fstab was not updated, but even if it were, I don't
   see how adding this line to /etc/fstab would actually mount it, there
   should still be the actual "mount /cache_for_lmi" command somewhere
   and it just doesn't seem to be present anywhere.

1. "mkdir --parent /cache_for_lmi/downloads" in the beginning of
   install_msw.sh created C:/cygwin-lmi/cache_for_lmi/downloads
   because /cache_for_lmi hadn't been mounted on C:/cache_for_lmi initially.

   And this is indeed confirmed by this snippet from the beginning of the
   log:

        mount
        C:/cygwin-lmi/bin on /usr/bin type ntfs (binary,auto)
        C:/cygwin-lmi/lib on /usr/lib type ntfs (binary,auto)
        C:/cygwin-lmi on / type ntfs (binary,auto)
        C: on /cygdrive/c type ntfs (binary,posix=0,user,noumount,auto)

2. Then install_msw.sh proceeded to mount /cache_for_lmi, from the log:

        mount --force "C:/cache_for_lmi" "/cache_for_lmi"

3. Then "make -f install_mingw.make" failed because now
   /cache_for_lmi/downloads maps to (non-existent) C:/cache_for_lmi/downloads
   directory.

4. Then "make -f install_mpatrol.make" created that directory and
   subsequent downloads were successful, but building everything failed
   because of lack of the compiler.


 Sorry for failing to understand it on my own and thanks for your help with
debugging it.

 There are several ways to fix this, the two most direct ones I see are:

(A) Move "mkdir --parents /cache_for_lmi/downloads" command below the
    mount commands in install_msw.sh. This is the simplest and most minimal
    solution.

(B) Remove mkdir from install_msw.sh entirely and let all makefiles
    downloading something create the directories they need, like
    install_mpatrol.make already does. This would arguably be the most
    robust solution. I'd also extract some common rules from all
    install_*.make into some common makefile which would be included by
    all of them to ensure that they all behave consistently.

 Should I do either (A) or (B) or do you prefer to do these changes
yourself?


GC> >  There is also clearly a problem with detecting the script success. IMO it
GC> > shouldn't continue running if any of the steps failed, i.e. I'd just add
GC> > "set -e" to the top of it. But if this is too extreme, it should at least
GC> > exit with a clear error message if installing the compiler fails,
GC> > continuing in this case just doesn't make any sense.
GC> 
GC> That's deliberate. Maybe not ideal, but deliberate. Perhaps there are
GC> some commands whose failure may be tolerable, so that "set -e" would
GC> cause the script to fail when it would otherwise work acceptably.

 I understand but disagree. IMHO it's better to explicitly deal with
commands whose failure is acceptable rather than suppose that failure is
harmless by default, which is a dangerous, and generally wrong, assumption
to make.

 Regards,
VZ