Re: [lmi] Cygwin build failure

[Vadim Zeitlin]

On Tue, 30 Apr 2019 15:43:47 +0000 Greg Chicares <address@hidden> wrote:

GC> I suspect that the anomaly stems from one of three causes: either
GC>  - the '7-Zip' command doesn't do what 'bsdtar' does, and you're
GC>    using the '7-Zip' command; or

 7z does do the same thing as bsdtar does, I've checked this. But in any
case, I saw the problem even running bsdtar manually and the absence of any
7z output in the log shows that the script used bsdtar as well.

GC>  - cygwin 'bsdtar' doesn't do what debian 'bsdtar' does, which
GC>    seems unthinkable; or
GC>  - the 'cp' and 'mv' commands don't do the same thing.

 They do the same thing, but both of them behave differently depending on
whether the destination directory exists or not and this is, of course, the
explanation, I feel pretty stupid for not realizing it yesterday.

GC> I'm often surprised by the result of copying directories with 'cp':
GC> sometimes the contents are copied, and sometimes the directory (with
GC> its contents) is copied instead. I've never found a good explanation.

 If the target directory exists, the source directory is copied into it. If
it doesn't exist, then source directory is "renamed into" it.

GC> Might 'cp' or 'mv' do different things depending on whether or not the
GC> destination directory already exists?

 Emphatically yes.

GC> Do you know a way to write those commands that would avoid any
GC> ambiguity?

 I don't know of any other way than ensuring that the target directory
[does not] exist[s] before running the command.

GC> Can you say which command ('bsdtar' or '7-Zip') is actually performing
GC> the extraction on your machine, and exactly what it does?

 bsdtar is used and it works exactly as expected.

GC> And here's the thing about 'cp' that confuses me, which looks like
GC> it might explain the anomaly you report:
GC> 
GC> /opt/lmi/src/lmi[0]$rm -rf /opt/lmi/does_not_exist
GC> /opt/lmi/src/lmi[0]$mkdir /opt/lmi/exists
GC> /opt/lmi/src/lmi[0]$
GC> /opt/lmi/src/lmi[0]$cp --archive /tmp/mingw8-ad_hoc/mingw32 
/opt/lmi/does_not_exist 
GC> /opt/lmi/src/lmi[0]$ls -o 
/opt/lmi/does_not_exist/bin/i686-w64-mingw32-gcc-8.1.0.exe 
GC> -rwxr-xr-x 1 greg 1862656 May 12  2018 
/opt/lmi/does_not_exist/bin/i686-w64-mingw32-gcc-8.1.0.exe
GC> /opt/lmi/src/lmi[0]$
GC> /opt/lmi/src/lmi[0]$cp --archive /tmp/mingw8-ad_hoc/mingw32 /opt/lmi/exists 
        
GC> /opt/lmi/src/lmi[0]$ls -o /opt/lmi/exists/bin/i686-w64-mingw32-gcc-8.1.0.exe
GC> ls: cannot access '/opt/lmi/exists/bin/i686-w64-mingw32-gcc-8.1.0.exe': No 
such file or directory
GC> /opt/lmi/src/lmi[2]$ls -o 
/opt/lmi/exists/mingw32/bin/i686-w64-mingw32-gcc-8.1.0.exe 
GC> -rwxr-xr-x 1 greg 1862656 May 12  2018 
/opt/lmi/exists/mingw32/bin/i686-w64-mingw32-gcc-8.1.0.exe

 This is indeed the explanation and it also shows that the commit which
really broke this is very recent, which explains why the problem hasn't
been detected yet: it's the innocently-looking 9afb0922 from 2019-04-14
which renamed "scratch" directories -- but while renaming them, it also
made the "ad_hoc" directory (BTW, I think "scratch" was a more descriptive
name, I wonder why did it have to be changed) a subdirectory of $prefix,
which resulted in the observed problem.

 You surely see it yourself already, but, just to dissipate any lingering
uncertainty, the problem is that the "initial_setup" target does

        $(MKDIR) --parents $(ad_hoc_dir)

and ad_hoc_dir is now /MinGW_/ad_hoc, so creating it creates /MinGW_ as
well, which mean that "$(CP) --archive $(ad_hoc_dir)/mingw32 $(prefix)"
command executed soon after it copies mingw32 to an already existing
$(prefix) and so results in creating $(prefix)/mingw32.

 As an aside, this works in your Linux chroot because /opt/lmi/MinGW_8
doesn't exist when "cp --archive /tmp/mingw8-ad_hoc/mingw32
/opt/lmi/MinGW_8" command is executed.


GC> Bonus question: what does
GC>     $(MKDIR) --parents $(prefix)
GC>     $(RM) --force --recursive $(prefix)
GC> in that makefile do? I'm not sure, but I suppose mkdir is used
GC> in an attempt to make the rm command safer. If this is a good
GC> idiom, then how should we document it so that I can understand it?
GC> Otherwise, what should we replace it with?

 This makefile is just very confusing in general as it tests for clearly
impossible things, e.g. it checks for ad_hoc_dir existence after ensuring
that its parent directory does _not_ exist which is useless unless you're
trying to protect against some race condition but if you are, then it's
still useless, of course, because the directory might be created just after
this check. And creating the directory before erasing it seems to fall into
the same utility category.

 But I guess that the former might be a leftover from the time when
ad_hoc_dir (née scratch_dir) wasn't a subdirectory of the prefix directory.
For the latter I just don't have any explanation at all.


 Anyhow, the important question is how do we fix this? The two obvious
solutions I see are:

1. Change the copy command from

        $(CP) --archive $(ad_hoc_dir)/mingw32 $(prefix)

   to

        $(CP) --archive $(ad_hoc_dir)/mingw32/* $(prefix)

   (notice that the latter command will fail if $(prefix) doesn't exist,
   which is a good thing).

2. Change ad_hoc_dir to not be a subdirectory of $(prefix) (in this case
   we probably should also add a check that $(prefix) doesn't exist, to
   avoid running into the same problem again later).

 I can't think of anything else right now, but IMO (1) doesn't really have
any drawbacks, so why not just do it?

 Of course, I'd still prefer to remove the useless checks and commands from
the "initial_setup" target, but this is independent of the issue at hand.

 Regards,
VZ