bug#10471: Severe or critical - deletes existing files and leaves nothing. (cp)

[L. A. Walsh]

reopen 10468
thanks
=============

 Not to stir up problems, but those who do not learn from history are
doomed to repeat it.  Nothing in this re-opening should be
taken personally, but am trying to follow-up on a bug that, I
felt, was improperly closed, and that now looks like it may be
'touched' or changed by a new fix to 'mv'.  The description in
the git-coreutils (not yet released) matches symptoms almost
exactly as I described over 3 years ago in 'cp'. 

 Since 'cp', 'ln' and 'mv' are closely related, and since 10468/10471
were still unfixed in coreutils 8.21-7.1.3 (from suse 13.1) as that
bug 10468 was still unfixed in 8.21-7.1.3 (from OpenSuse 13.1),
on linux w/XFS (as was described in this bug's original follow-up),
I was wondering if the changes to 'mv' might also have fixed this
(10468/10471) bug.  As the comment in the NEWS file made no mention of
behavior changes to 'cp' or either of these bug #'s being fixed,
it wasn't clear if the new work on 'mv' also fixed the similar (if not 
identical) problems reported against 'cp'.


 From the NEWS file in the *current* coreutils git:


  mv no longer supports moving a file to a hardlink, instead issuing
  an error.  The implementation was susceptible to races in the
  presence of multiple mv instances, which could result in both
  hardlinks being deleted.  Also on case insensitive file systems
  like HFS, mv would just remove a hardlinked 'file' if called like
  `mv file File`.  The feature was added in coreutils-5.0.1.


Please note, as reported below that this problem was reported in
behavior of "cp -au" over 3 years ago and was shut out as being
a cygwin-only bug. 

I told the maintainer (at that time) how to reproduce it on linux & XFS.
Last I checked, ~6 months ago, it was still present on linux w/XFS as I'd
described). 

I was told to "trust the coreutils maintainers"(1) in the closing of
what I thought was a serious bug" -- even though also said it could be
reproduced on linux-only w/XFS (as it was among the 1st with an ignore
case option).  Nevertheless, the bug was dropped in a crack as I said
would happen if the bug was 'closed out' as 'not-a-bug'.  I tried to
emphasize the increasing impact of this bug as more file systems
implemented case-preservation+insensitivity (ZFS and MacOS's FS, can also
be added to the group having such an option).
 
(1) -------- Original Message --------
  Subject: Re: bug#10471: closed (Severe or critical -
           deletes existing files and leaves nothing. (cp))
  Date: Mon, 09 Jan 2012 20:59:36 -0500
  From: Glenn

  I'm saying you should trust the coreutils maintainers to reach the
  right decision.

  Anyway, this mailing list is for technical issues with
  debbugs.gnu.org.  It's not the place to discuss how the various
  projects that use it choose to do so.


-------------------

Given the comment in the NEWS file  -- that the behavior change was in
'mv', I'm wondering if this was also fixes the same problem in 'cp' or if
that is still in the "closed-in-coreutils-as-a-cygwin-bug" category.
Below is the original bug that shows examples of the upper and lower case
hard-linked filenames causing mutually assured destruction (with 'cp' both
w+w/o preserve-links & update) -- which I thought (and still do think) was
worse than the 'mv' case because of the "-u" flag -- which I thought (at
that time) meant only update those files that were newer or didn't exist
on the target.  But it didn't:

I had 'Aaa' (v2) on source, and something like 'Aaa' (v1) <=> 'aaa' (v1)
on the destination (the ' (v1)'/' (v2)' not being part of the name, but to
indicate relative time stamps).  It appeared (or one explanation was),
that:

1) src(Aaa) removed the linked 'aaa' and 'Aaa' files at dst & copied (Aaa)
2) src then tried to link 'aaa'->'Aaa' on dst, but at link time, both
copies at the dst were deleted so we saw msg 'cannot create hard link: 
ENOENT.
Result) -- both copies of the older file deleted on the dst.  i

There may have been some variations on this, I'm not familiar with the
exact algorithm used at the time.  I noted, however that in the NEWS file
under bug fixes, in the new coreutils that had recently made it to cygwin
(release 8.13 (2011-09-08) (...cont...)

  cp -u -p would fail to preserve one hard link for each up-to-date
  copy of a src-hard-linked name in the destination tree.  I.e., if
  s/a and s/b are hard-linked and dst/s/a is up to date, "cp -up
  s dst" would copy s/b to dst/s/b rather than simply linking
  dst/s/b to dst/s/a.

...and I suggested that this new bug might be related to that.

Regardless. It was closed out and fell into the cracks.  Now a related
bug in 'mv' (which shares much code with cp, I believe), is reported as
being 'fixed'...   So I'm wondering if the bug I reported was fixed as
a result of this, or if it is still there, or what?

Original bug response and a response about categorizing this bug as
a windows-only bug, being 'impractical' to look at in coreutils and
having near zero chance of being fixed via coreutils, attached
below.  I included it for reference, since I was overruled, at the time
by those who thought the bug was *only in cygwin* (even though I
did claim in followup discussion that it could also be reproduced
on linux w/XFS using its '-n version=ci' mkfs option.  It can probably
also be reproduced, now, on ZFS and some MacOS FS's that have similar
case-treatment options.

Cheers -- and still wondering if the 8.13 change was related to this
bug and if the new 'mv' behavior change affects this 'cp' bug?

-- Linda Walsh



-------- Original Message --------
Subject: Re: bug#10468: BUG: Severe or critical -
--------     deletes existing files and leaves nothing. (cp)

Date: Mon, 09 Jan 2012 14:41:22 -0700
From: Eric
Organization: Red Hat
To: Linda
CC: 10468-debbugs-gnu-org, "cygwin"
References: <address@hidden>

tag 10468 notabug
thanks

On 01/09/2012 02:19 PM, Linda,  wrote:
>  I was trying to copy a font dir from a unix to a windows machine.
>
>  Problem is there are duplicates -- where only the case differs...

The problem is not in coreutils, but in your operating system's
limitations, and in your configuration.  Windows (and thus Cygwin) can
be put in a mode where it preserves case (and I highly recommend doing
so if you want your example to work):
http://cygwin.com/cygwin-ug-net/using-specialnames.html#pathnames-casesensitive

Without that setting, there is nothing that coreutils can do on your
behalf: the default case-insensitive behavior of Windows violates POSIX,
and forcing coreutils to add bloat to work around this violation is
rather difficult and unpalatable compared to all other operating systems
that already follow POSIX.  So I am closing the upstream coreutils bug
report.

There may be a way to make the cygwin port of coreutils smarter about
case-sensitive issues when the Windows setting is case-insensitive, but
it would be specific to the cygwin port.  Meanwhile, I'm not convinced
it is worth slowing down the common case by checking for case clash on
all operations, since most users that either avoid case clash or have
the registry setting on, just to help out the rare case of a user that
has case clash but the registry setting off.  You are more than welcome
to discuss this further on the cygwin list, and preferably provide
patches if you want behavior changed, but again, that would most likely
be cygwin-specific and does not need to involve the upstream coreutils
list (the cygwin port of coreutils already carries several other patches
for dealing with non-POSIX issues that don't need to be ported back
upstream, such as how to handle .exe suffixes).

> > cp -rvu /usr/share/fonts> cp -rvu //bliss/usr_share/fonts/. .
>
>  figuring without the "-a" it wouldn't try to force linking, thus no
>  prob... ... well...
>  removed `././OTF/AJensonPro-Bold.otf'
>  cp: cannot create hard link `././OTF/AJensonPro-Bold.otf' to
>  `././OTF/ajensonpro-bold.otf': No such file or directory
>  removed `././OTF/AJensonPro-BoldCapt.otf'
>  cp: cannot create hard link `././OTF/AJensonPro-BoldCapt.otf' to
>  `././OTF/ajensonpro-boldcapt.otf': No such file or directory
>  removed `././OTF/AJensonPro-BoldDisp.otf'
>  ------
>  ?!!?!   Why is it trying to create hard links again?  I didn't specify
>  preserve links!?!
>  (i.e. this is a cp BUG...)

If I understand correctly, your problem stems from the fact that your
source location has multiple hard links to the same file but with
different case, while your destination can't support two hard links to
the same file differing only in case, so there is no sane way to say
which of the two spellings should be copied.  It's not cp's fault that
you are copying from a permissive system to a restrictive one; and while
the errors may not be the most friendly, at least they are accurate.


-------- Original Message --------
Subject: Re: bug#10468: BUG: Severe or critical -
--------     deletes existing files and leaves nothing. (cp)
Date: Mon, 09 Jan 2012 17:15:26 -0800
From: Paul
Organization: UCLA Computer Science Department
To: debbugs-gnu-org, tlinx-org

On 01/09/12 17:00, Linda Walsh wrote:
>  cp is a gnu util, not part of linux, -- you should
>  demonstrate that it is not a bug in cygwin

This is not a practical division of labor.  As a general rule,
we don't have the resources to deal with Windows ports.

I suggest raising this issue with the people who are actually
doing the Windows port, whoever they are.  If it's a
problem with the port, they can fix it themselves without putting
further load on us; if not, they'll have a better chance to diagnose
and fix the problem than we will (our chance is essentially zero).