Re: Is there a memory leak in find ?

[Bernhard Voelker]

On 04/11/2014 06:28 AM, Paul E Condon wrote:
> I've done more testing using several different trees as input data. In
> all cases the script terminates well before it has completely walked
> the tree, always when the output file has grown to about 2.8GB. The
> last line is always a complete well formed output line with no sign of
> syserr stuff.  The project that I'm working on is my own variation of
> using rsync to do frequent snapshot backups along the lines discussed
> by Mike Rubel in
> 
> http://www.mikerubel.org/computers/rsync_snapshots/
> 
> In my (growing) backup tree(s) there are many files with hundreds of
> hard links. Find does nothing more the report the numerical value of
> the hard link count, so I don't think the hard links are the source of
> the problem.  But they are a reason that the problem needs to be
> solved. But first ...  you must be able to duplicate the problem.

I'm also using my own rsync-based snapshooting - with 24x hourly +
7x daily + 4x weekly + 12x monthly + 1x additional "LATEST" snapshots.
That makes e.g. 48 hardlinks e.g. for my ~/.profile, and every snapshot
has about ~87000 files for 67G of data:

  $ ls -ldog home/daily.0/berny/.profile
  -rw-r--r-- 48 1149 Feb 15  2013 home/daily.0/berny/.profile

  $find home/daily.0 | wc -l
  87919

  $ du -shx home/daily.0
  67G     home/daily.0

Which version of 'find' are you using?

I don't see this mem-leaking effect - neither with latest 'find' from
the git repository nor with that on my openSUSE-13.1 system (4.5.12).
This is what /usr/bin/time reports:

        User time (seconds): 18.55
        System time (seconds): 24.71
        Percent of CPU this job got: 99%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:43.58
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 11040
        Average resident set size (kbytes): 0

Looking at the git history, and if your find verison is <=  4.5.10,
then I'd guess you hit the bug fixed with this commit

  http://git.sv.gnu.org/cgit/findutils.git/commit/?id=e17d58833b

  find: fix two time-formatting leaks (bug #37356)

  * find/print.c (do_time_format): Call xmalloc for static "buf" only
  the first time.
  When reallocating buf, be sure to update its buf_size.
  Also free "altbuf".
  Reported by Nemo Maelstrom Thorx in http://bugs.debian.org/687358
  via Andreas Metzler in http://savannah.gnu.org/bugs/?37356

If this is not the case, then please tell us more about how to
produce the OOM.

Meanwhile, and if this is problem really originates from the massive
hard-linking, then you could probably work around it by calling the
find(1) for every single snapshot, and prefixing the last column with
the snapshot name ("hourly.0/" etc.) yourself.

Have a nice day,
Berny