bug-fileutils
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

coreutils-4.5.8 released


From: Jim Meyering
Subject: coreutils-4.5.8 released
Date: Sat, 22 Feb 2003 08:41:17 +0100

The big news is that du works better now.
I broke it in 4.5.7.  So much for my ego.

I've added some basic tests of du that actually compare block counts
on at least some systems, so such a regression should not recur.
Thanks to Bruno Haible for reporting the problem and helping clean
up the affected code.

But I'm still not done with du.
This began some time ago when I realized that the presence of
a very deep hierarchy (on the order of 25,000-30,000 levels)
could cause du to segfault due to its recursion exceeding
a system's stack size limit.  Using nftw didn't solve that
problem, since it too is implemented with explicit recursion.
Lots of programs have this limitation.

One proposed work-around was to use makecontext/setcontext to run
the offending code in a context with a very large, (mmap'd) stack.
I've implemented that, and although it does work -- at least on Linux --
the mmap-stack.[ch] business is obviously just a band-aid.
I expect to switch du.c to use fts soon, since it

  http://www.hmug.org/man/3/fts.html

isn't hindered by explicit recursion as ftw is, and since
it already has all of the functionality required by du.
I've already implemented a version of du.c to use glibc's fts.c,
and did some informal performance testing.  The current nftw-based
implementation is usually faster, but also less robust.  There are
always pros and cons:

  * any explicitly recursive implementation (glibc's ftw.c) is subject
    to *some* limitation on stack depth

  * an fts-based implementation performs about double the number of
    `stat' calls per directory to keep from being tricked by the renaming
    of a sub-hierarchy in which fts is currently operating.
    An (n)ftw-based implementation can be tricked -- and as you might
    recall, before those checks were added to GNU rm, opportunities for
    such trickery can lead to nasty security problems.
    I might give fts an option to disable the additional checks,
    because in most cases, there is no risk of trickery.

  * fts does more bookkeeping and has a bigger memory footprint

Initially I wrote the following entry for NEWS, but am unhappy
with the fact that the implementation currently works only
with Linux, so have disabled it.  If you want to experiment,
remove the #undef from lib/mmap-stack.h, but beware that it
mmap's 250MB.  Something about all of this would make cp segfault
all the time on at least one type of Solaris system.

  * du can now process much deeper hierarchies on some systems (subject to
    mmap support, size of virtual memory, etc.).  Before, it would segfault
    on hierarchies deeper than 20,000-30,000 levels due to stack limitation
    and excessive recursion.

Jim

P.S. I am looking for a new job, so as you might guess,
would appreciate any pointers to interesting opportunities.


  ftp://alpha.gnu.org/gnu/coreutils/coreutils-4.5.8.tar.gz   (5.8MB)
  ftp://alpha.gnu.org/gnu/coreutils/coreutils-4.5.8.tar.bz2  (3.8MB)
  http://fetish.sf.net/coreutils-4.5.8.tar.gz   (5.8MB)
  http://fetish.sf.net/coreutils-4.5.8.tar.bz2  (3.8MB)

And here are xdelta-style diffs:
  ftp://alpha.gnu.org/gnu/coreutils/coreutils-4.5.7-4.5.8.xdelta   (420KB)
  http://fetish.sf.net/coreutils-4.5.7-4.5.8.xdelta   (420KB)

Here are GPG detached signatures:
  ftp://alpha.gnu.org/gnu/coreutils/coreutils-4.5.8.tar.gz.sig
  ftp://alpha.gnu.org/gnu/coreutils/coreutils-4.5.8.tar.bz2.sig
  http://fetish.sf.net/coreutils-4.5.8.tar.gz.sig
  http://fetish.sf.net/coreutils-4.5.8.tar.bz2.sig

Here are the MD5 and SHA1 signatures:

66b5fec38d0d853101476adef46a6025  coreutils-4.5.8.tar.gz
c2855215753d5dcf4aff69f7dcb4b9a5  coreutils-4.5.8.tar.bz2
2cc9fd115ccb3a2a6951120ed8277c23  coreutils-4.5.7-4.5.8.xdelta
d0ec343b9b31a4f08a624b6881dcc4d4c6fb550e  coreutils-4.5.8.tar.gz
c942e0c39ca84b7a5df7549d09d876e65963c15c  coreutils-4.5.8.tar.bz2
529a836c4862e107099dbdba9555e001d45623f2  coreutils-4.5.7-4.5.8.xdelta
NEWS:
* du -S once again works like it did before the change in 4.5.5
* stat accepts a new file format, %B, for the size of each block reported by %b
* du accepts new option: --apparent-size
* du --bytes (-b) works the same way it did in fileutils-3.16 and before
* du reports proper sizes for directories (not zero) (broken in 4.5.6 or 4.5.7)
* df now always displays under `Filesystem', the device file name
  corresponding to the listed mount point.  Before, for a block- or character-
  special file command line argument, df would display that argument.  E.g.,
  `df /dev/hda' would list `/dev/hda' as the `Filesystem', rather than say
  /dev/hda3 (the device on which `/' is mounted), as it does now.
* test now works properly when invoked from a set user ID or set group ID
  context and when testing access to files subject to alternate protection
  mechanisms.  For example, without this change, a set-UID program that invoked
  `test -w F' (to see if F is writable) could mistakenly report that it *was*
  writable, even though F was on a read-only file system, or F had an ACL
  prohibiting write access, or F was marked as immutable.


ChangeLog entries:

**********************************************************************
ChangeLog       21 Feb 2003 20:38:16 -0000      1.246
**********************************************************************
2003-02-21  Jim Meyering  <address@hidden>

        * Version 4.5.8.

        Merge in changes from autoconf's version of this file.
        * Makefile.maint (www-gnu): Define.
        (standards.texi-url_prefix): Use $(www-gnu).
        (make-stds.texi-url_prefix): Likewise.

        * src/cp.c: Include "mmap-stack.h".
        (main): Invoke `run' through a macro that (when possible) runs it
        with a large, mmap'd stack.

        * src/cp.c (run): New function, preparing for the above.
        Exit from this function, not from main
        (main): Call run.

        * src/du.c: New option: --apparent-size.
        (enum) [APPARENT_SIZE_OPTION]: New member.
        (long_options): Add it.
        (usage): Describe it.
        (main): Handle it.
        ['b']: Set apparent_size.
        David Eisner reported that the behavior of --bytes had changed.
        Paul Eggert proposed the use of a new option, --apparent-size.

        * src/du.c (apparent_size): New global.
        (print_only_size): Reflect the fact that we're printing byte counts,
        not ST_NBLOCKSIZE-byte-block counts.
        (print_size): Call print_only_size rather than duplicating its code.
        (process_file): Accumulate byte counts, rather than block counts.

        * src/du.c (process_file): Always reset size_to_propagate_to_parent
        for --separate-dirs (-S).

2003-02-20  Jim Meyering  <address@hidden>

        * Use automake-1.7.3.  Regenerate dependent files.

        * src/stat.c (print_stat): New format: %B (to print ST_NBLOCKSIZE).
        This makes %b (number of ST_NBLOCKSIZE-byte blocks) more useful.
        (usage) [%B]: Describe it.
        [%b]: Refer to %B.

        * src/du.c (process_file): Reorganize the code to use only
        one `sum' array, and change how -S works back to the way it was
        before 2003-01-31.  Patch by Bruno Haible.

        * tests/du/basic: New test.
        * tests/du/Makefile.am (TESTS): Add basic.

        * tests/envvar-check: Add checks for the following:
        BLOCK_SIZE, DU_BLOCK_SIZE, DF_BLOCK_SIZE, LS_BLOCK_SIZE.

        * tests/Makefile.am: Rename phony target envvar-check to evar-check
        so as not to conflict with the distributed file by the same name.

        * src/du.c (process_file): Set info->skip before any possible return.

        Report correct usage for directories, not 0.
        * src/du.c (process_file): Return for `file_type == FTW_DPRE'
        _before_ recording the dev/ino of a directory.
        Reported by Bruno Haible.

        Now, df always displays the device file name corresponding to the
        listed mount point under `Filesystem'.  Before, for an unmounted
        block- or character-special file argument, it would display the
        command-line argument instead.
        * src/df.c (show_disk): Return a value indicating whether
        there was a match.  Don't try to find a mount point here.
        (show_entry): If show_disk doesn't find a match, call show_point.

2003-02-19  Jim Meyering  <address@hidden>

        * src/du.c: Include "mmap-stack.h".
        (du_files): Add prototype with ATTRIBUTE_NORETURN.
        Exit from this function, not from...
        (main): ...here.
        Instead, if possible, invoke du_files through a macro that
        runs it with a large, mmap'd stack.

        * src/join.c (usage): Change wording in --help output:
        use FILENUM instead of `SIDE' and say what FILENUM means.
        Reported by Bernhard Gabler.

        * src/df.c (print_header): Rather than using a hard-coded literal
        string of spaces matching the length of the English `...Type' header,
        output the right number of spaces to match the selected translation.
        Reported by Yann Dirson and Jean Charles Delepine as Debian bug 131113.

        * src/split.c (bytes_split): Remove unnecessary `else' after break.
        (lines_split): Likewise.  and correct misleading indentation.

        * src/split.c: Include "full-read.h".
        (bytes_split, lines_split, line_bytes_split): Use full_read,
        not safe_read.   The way split was using the latter, a short read
        could cause split to terminate before EOF.

        * tests/misc/tty-eof: Test all programs that can read stdin,
        requiring no arguments and that write to standard output.

        * tests/misc/tty-eof: New file.  Renamed from ...
        * tests/misc/cat-tty-eof: Remove file.  Rename to tty-eof.
        * tests/misc/Makefile.am (TESTS): Reflect renaming.

2003-02-18  Jim Meyering  <address@hidden>

        cksum would perform an extra read after encountering EOF
        * src/cksum.c (cksum): Exit the loop upon EOF, too.
        Patch by Michael Bacarella.

        Test for the bug fixed today in cksum, md5sum, and sha1sum.
        * tests/misc/cat-tty-eof: Generalize, clean-up, and test for
        cat, cksum, md5sum, and sha1sum all in the same loop.

2003-02-14  Jim Meyering  <address@hidden>

        * src/remove.c: Include "euidaccess.h".
        Remove declaration of euidaccess.

2003-02-12  Jim Meyering  <address@hidden>

        * src/pathchk.c (portable_chars_only): Remove unnecessary `const'
        in cast to avoid warning from icc.  Reported by Alexandre Duret-Lutz.

2003-02-10  Jim Meyering  <address@hidden>

        * src/test.c: Don't include group-member.h.
        Include euidaccess.h.
        (eaccess): Rewrite function to set the real uid and gid temporarily
        to the effective uid and gid, then invoke 'access', and then set the
        real uid and gid back.  On systems that lack setreuid or setregid,
        fall back on the kludges in euidaccess.  Before, it would not work
        for e.g., files with ACLs, files that were marked immutable,
        or on file systems mounted read-only.  Nelson Beebe raised the issue.
        Paul Eggert suggested the new implementation.

2003-02-09  Jim Meyering  <address@hidden>

        * src/test.c (test_stat): Remove function.  It's job is done (only
        when necessary) by the wrapper in lib/stat.c.  Adjust all uses.


**********************************************************************
lib/ChangeLog   21 Feb 2003 17:13:17 -0000      1.497
**********************************************************************
2003-02-21  Jim Meyering  <address@hidden>

        * mmap-stack.h: Undefine HAVE_MMAP_STACK, thus disabling
        this code on all systems.

        Merge in some changes from GNU libc.
        * md5.h (md5_uintptr): Define.

        * ftw.c: Include <sys/types.h> before <dirent.h>.
        This is required for Apple Darwin 6.3 (MacOS 10.2.3).
        Patch by Nelson Beebe.

2003-02-19  Jim Meyering  <address@hidden>

        * md5.c: Merge in some clean-up and optimization changes from glibc.
        * sha.c: Apply those same changes here, too.

2003-02-18  Jim Meyering  <address@hidden>

        * md5.c (md5_stream) [BLOCKSIZE]: Move definition to top of file.
        Ensure that it is a multiple of 64.
        Rearrange loop exit tests so as to avoid performing an
        additional fread after encountering an error or EOF.
        * sha.c (sha_stream): Likewise.
        Reported by Michael Bacarella.

        * Makefile.am (CLEANFILES): Remove ftw.h and search.h.
        (DISTCLEANFILES): Add them here.
        Add fnmatch.h, too.

        * sha.c (sha_stream): Rearrange loop exit tests so as to avoid
        performing an additional fread after encountering an error or EOF.

2003-02-17  Jim Meyering  <address@hidden>

        * mmap-stack.h (RUN_WITH_BIG_STACK_4): Define.

        * physmem.c: Undo most of last change.
        Use sysmp instead, since it provides a cleaner interface.
        From Kaveh Ghazi.

2003-02-16  Jim Meyering  <address@hidden>

        * Makefile.am (libfetish_a_SOURCES): Add mmap-stack.h.

        * mmap-stack.c (MAP_ANONYMOUS): Use MAP_ANON, if possible.
        Initialize `fd' to -1, so Solaris' mmap works with MAP_ANON.

2003-02-15  Jim Meyering  <address@hidden>

        * mmap-stack.c, mmap-stack.h: New files.
        Let the caller run a function with a larger (mmap'd) stack.

        Add Irix6 support to physmem.c.
        * physmem.c (irix_sysget): New function.
        (IRIX_SYSGET_TOTAL, IRIX_SYSGET_AVAILABLE): New macros.
        (physmem_total, physmem_available): Use them.
        (main) [DEBUG]: New function.

2003-02-11  Jim Meyering  <address@hidden>

        * Makefile.am (libfetish_a_SOURCES): Add euidaccess.h.
        * euidaccess.c: Include "euidaccess.h".
        * euidaccess.h: New file.


**********************************************************************
m4/ChangeLog    19 Feb 2003 08:40:10 -0000      1.464
**********************************************************************
2003-02-19  Jim Meyering  <address@hidden>

        * c-stack.m4 (AC_SYS_XSI_STACK_OVERFLOW_HEURISTIC): Limit stack size
        to 1MB, so as not to render systems with no stack size limit (e.g.,
        linux-2.2.x) unusable.  Suggestion and code from Bruno Haible.

2003-02-17  Jim Meyering  <address@hidden>

        * prereq.m4 (jm_PREREQ_PHYSMEM): Undo last change, since
        Kaveh Ghazi found a better way to get the required information.
        Add check for sys/sysmp.h.

2003-02-15  Jim Meyering  <address@hidden>

        * mmap-stack.m4: New file.

        * jm-macros.m4: Require AC_SYS_MMAP_STACK.

        Add Irix6 support to physmem.c.
        * prereq.m4 (jm_PREREQ_PHYSMEM): Also check for sys/sysget.h
        and sys/sysinfo.h.
        Also check for sysget.
        Reported by Kaveh Ghazi.

2003-02-12  Jim Meyering  <address@hidden>

        * restrict.m4 (ACX_C_RESTRICT): Remove #ifndef -- so now this
        macro also checks for support when using a C++ compiler.
        Also, remove the test for SGI's __restrict.
        Suggested by Steven G. Johnson.

        * regex.m4 (jm_PREREQ_REGEX): Require ACX_C_RESTRICT.

        * restrict.m4 (ACX_C_RESTRICT): Minor syntactic changes:
        Split long lines, use AC_COMPILE_IFELSE, indent, use `case'
        instead of nested `if's, remove unnecessary quotes.

        * restrict.m4 (ACX_C_RESTRICT): New macro.
        Copied directly from the URL in the comments.
        By Steven G. Johnson.

2003-02-09  Jim Meyering  <address@hidden>

        * check-decl.m4 (jm_CHECK_DECLS): Add euidaccess.


**********************************************************************
po/ChangeLog    16 Feb 2003 21:00:07 -0000      1.11
**********************************************************************
2003-02-16  Jim Meyering  <address@hidden>

        * LINGUAS: Add Finnish (fi).


**********************************************************************
doc/ChangeLog   21 Feb 2003 12:24:32 -0000      1.83
**********************************************************************
2003-02-21  Jim Meyering  <address@hidden>

        * coreutils.texi (du invocation): Document --apparent-size.
        Adjust documentation of --bytes (-b).
        (stat invocation): Describe %B.

Attachment: pgpUvMWRL9Lrv.pgp
Description: PGP signature


reply via email to

[Prev in Thread] Current Thread [Next in Thread]