Re: Is there a memory leak in find ?

[Paul E Condon]

On 20140411_0830+0200, Bernhard Voelker wrote:
> On 04/11/2014 06:28 AM, Paul E Condon wrote:
> > I've done more testing using several different trees as input data. In
> > all cases the script terminates well before it has completely walked
> > the tree, always when the output file has grown to about 2.8GB. The
> > last line is always a complete well formed output line with no sign of
> > syserr stuff.  The project that I'm working on is my own variation of
> > using rsync to do frequent snapshot backups along the lines discussed
> > by Mike Rubel in
> > 
> > http://www.mikerubel.org/computers/rsync_snapshots/
> > 
> > In my (growing) backup tree(s) there are many files with hundreds of
> > hard links. Find does nothing more the report the numerical value of
> > the hard link count, so I don't think the hard links are the source of
> > the problem.  But they are a reason that the problem needs to be
> > solved. But first ...  you must be able to duplicate the problem.
> 
> I'm also using my own rsync-based snapshooting - with 24x hourly +
> 7x daily + 4x weekly + 12x monthly + 1x additional "LATEST" snapshots.
> That makes e.g. 48 hardlinks e.g. for my ~/.profile, and every snapshot
> has about ~87000 files for 67G of data:
> 
>   $ ls -ldog home/daily.0/berny/.profile
>   -rw-r--r-- 48 1149 Feb 15  2013 home/daily.0/berny/.profile
> 
>   $find home/daily.0 | wc -l
>   87919
> 
>   $ du -shx home/daily.0
>   67G     home/daily.0
> 
> Which version of 'find' are you using?
> 
> I don't see this mem-leaking effect - neither with latest 'find' from
> the git repository nor with that on my openSUSE-13.1 system (4.5.12).

I'm using 4.4.2 from Debian Wheezy. Your infomation is most useful, *and*
most encouraging. Now I go to debian-user list for advice on how to 
upgrade with least upset to the Debian life-style ;-)

Thankyou


> This is what /usr/bin/time reports:
> 
>         User time (seconds): 18.55
>         System time (seconds): 24.71
>         Percent of CPU this job got: 99%
>         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:43.58
>         Average shared text size (kbytes): 0
>         Average unshared data size (kbytes): 0
>         Average stack size (kbytes): 0
>         Average total size (kbytes): 0
>         Maximum resident set size (kbytes): 11040
>         Average resident set size (kbytes): 0
> 
> Looking at the git history, and if your find verison is <=  4.5.10,

Thankyou

> then I'd guess you hit the bug fixed with this commit
> 
>   http://git.sv.gnu.org/cgit/findutils.git/commit/?id=e17d58833b
> 
>   find: fix two time-formatting leaks (bug #37356)
> 
>   * find/print.c (do_time_format): Call xmalloc for static "buf" only
>   the first time.
>   When reallocating buf, be sure to update its buf_size.
>   Also free "altbuf".
>   Reported by Nemo Maelstrom Thorx in http://bugs.debian.org/687358
>   via Andreas Metzler in http://savannah.gnu.org/bugs/?37356
> 
> If this is not the case, then please tell us more about how to
> produce the OOM.
> 
> Meanwhile, and if this is problem really originates from the massive
> hard-linking, then you could probably work around it by calling the
> find(1) for every single snapshot, and prefixing the last column with
> the snapshot name ("hourly.0/" etc.) yourself.
> 
> Have a nice day,
> Berny
> 

Thankyou
This day is beginning very well, thankyou.

-- 
Paul E Condon           
address@hidden