Re: [bug #23023] cp -r use a lot of memory, feature or memory leak?

[Jim Meyering]

Matteo Boccafoli <address@hidden> wrote:
> URL:
>   <http://savannah.gnu.org/bugs/?23023>
>                  Summary: cp -r use a lot of memory,  feature or memory leak?
...
> # 21000 dir in 1.tar.bz2
> $ cd /dev/shm
> /dev/shm$ time tar -xjf 1.tar.bz2
>
> real    0m8.403s
> user    0m3.188s
> sys     0m5.196s
>
> /dev/shm$ time ./busybox cp -r /dev/shm/1/ /dev/shm/b
> real    0m54.421s
> user    0m0.832s
> sys     0m51.751s
> # [time t_1,t_2 : t_1<t_2 and cp.memory(t_1)==cp.memory(t_2)]
>
> /dev/shm$ time /bin/cp -r /dev/shm/1/ /dev/shm/c
> real    0m55.696s
> user    0m2.680s
> sys     0m50.447s
> #[time t_1,t_2 : t_1<t_2 and cp.memory(t_1)<cp.memory(t_2)]
> # using a lot of memory (max 54 MB of RAM)
>
> Is it a feature or memory leak?

As far as I know, cp has no memory leaks.
However, a peak memory usage of 54MB does sound high.
How did you measure it?

I know that "busybox cp" fails to preserve some hard-link
relationships that GNU cp does preserve.  That may explain
some of the difference.

Note however, that cp is far from optimal.
There is work underway to rewrite is using fts.c,
and that should reduce its memory consumption in addition
to making it more robust.

I tried to duplicate your situation as follows but ended up
with a peak memory usage of just 3.3 MB.

(BTW, how many *non-directories* are in your tarball?
 Count with this command: find 1 -type f|wc -l )

# Create a hierarchy of 21000 directories, each containing
# three empty files named a, b, and c.
$ ( rm -rf /tmp/1; mkdir /tmp/1 && cd /tmp/1 && seq 21000 \
    |xargs mkdir && seq 21000|xargs -i% echo %/a %/b %/c|xargs touch )

# run cp under valgrind's memory profiler
$ rm -rf massif.* /tmp/2; valgrind --tool=massif ./cp -r /tmp/1 /tmp/2

# display the first few lines of results (full output is over 2200 lines)
$ ms_print massif.out.*

--------------------------------------------------------------------------------
Command:            ./cp -r /tmp/1 /tmp/2
Massif arguments:   (none)
ms_print arguments: massif.out.30264
--------------------------------------------------------------------------------


    MB
3.273^                                                                     ,#
     |                                                                 ,@@@@#
     |                                                             .,@@@@@@@#
     |                                                         .. ::@@@@@@@@#
     |                                                       .@:: ::@@@@@@@@#
     |                                                  .. : :@:: ::@@@@@@@@#
     |                                               .: :: : :@:: ::@@@@@@@@#.
     |                                             , :: :: : :@:: ::@@@@@@@@#:
     |                                        ...@:@ :: :: : :@:: ::@@@@@@@@#:
     |                                    ..: :::@:@ :: :: : :@:: ::@@@@@@@@#:
     |                                  ..::: :::@:@ :: :: : :@:: ::@@@@@@@@#:
     |                           @   .@ ::::: :::@:@ :: :: : :@:: ::@@@@@@@@#:
     |                           @:: :@ ::::: :::@:@ :: :: : :@:: ::@@@@@@@@#:
     |                   ,   . : @:: :@ ::::: :::@:@ :: :: : :@:: ::@@@@@@@@#:
     |                   @: :: : @:: :@ ::::: :::@:@ :: :: : :@:: ::@@@@@@@@#:
     |               . ,@@: :: : @:: :@ ::::: :::@:@ :: :: : :@:: ::@@@@@@@@#:
     |            ,. : @@@: :: : @:: :@ ::::: :::@:@ :: :: : :@:: ::@@@@@@@@#:
     |        .. :@: : @@@: :: : @:: :@ ::::: :::@:@ :: :: : :@:: ::@@@@@@@@#:
     |     ,.@:: :@: : @@@: :: : @:: :@ ::::: :::@:@ :: :: : :@:: ::@@@@@@@@#:
     | .,@ @:@:: :@: : @@@: :: : @:: :@ ::::: :::@:@ :: :: : :@:: ::@@@@@@@@#:
   0 +----------------------------------------------------------------------->Mi
     0                                                                   155.8

Number of snapshots: 63
 Detailed snapshots: [2, 3, 4, 6, 10, 14, 15, 16, 21, 25, 34, 36, 43, 48, 50, 
51, 52, 53, 54, 55, 56, 57, 58, 59, 60 (peak)]