Re: overly aggressive memory usage by sort.c

[Paul Eggert]

On 06/11/2012 03:02 AM, Paul Eggert wrote:
> * src/sort.c (default_sort_size): Do not exceed 3/4 of total memory

Come to think of it, that patch is too conservative, as it
limits things to at most 3/8 of total memory.  The 3/4 limit
should be applied after 'size' is halved, not before.  Here's
a better patch:

>From dcfc5f8e6373dc7afe8e75c837dd81bbf724bb55 Mon Sep 17 00:00:00 2001
From: Paul Eggert <address@hidden>
Date: Mon, 11 Jun 2012 02:47:05 -0700
Subject: [PATCH] sort: by default, do not exceed 3/4 of physical memory

* src/sort.c (default_sort_size): Do not exceed 3/4 of total memory.
See Jeff Janes's bug report in
<http://lists.gnu.org/archive/html/coreutils/2012-06/msg00018.html>.
---
 src/sort.c |   11 ++++++++---
 1 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/src/sort.c b/src/sort.c
index 2593a2a..f21e092 100644
--- a/src/sort.c
+++ b/src/sort.c
@@ -1412,9 +1412,10 @@ default_sort_size (void)
   double mem = MAX (avail, total / 8);
   struct rlimit rlimit;
 
-  /* Let SIZE be MEM, but no more than the maximum object size or
-     system resource limits.  Don't bother to check for values like
-     RLIM_INFINITY since in practice they are not much less than SIZE_MAX.  */
+  /* Let SIZE be MEM, but no more than the maximum object size,
+     total memory, or system resource limits.  Don't bother to check
+     for values like RLIM_INFINITY since in practice they are not much
+     less than SIZE_MAX.  */
   size_t size = SIZE_MAX;
   if (getrlimit (RLIMIT_DATA, &rlimit) == 0 && rlimit.rlim_cur < size)
     size = rlimit.rlim_cur;
@@ -1434,6 +1435,10 @@ default_sort_size (void)
     size = rlimit.rlim_cur / 16 * 15;
 #endif
 
+  /* Similarly, leave a 1/4 margin for physical memory.  */
+  if (total * 0.75 < size)
+    size = total * 0.75;
+
   /* Return the minimum of MEM and SIZE, but no less than
      MIN_SORT_SIZE.  Avoid the MIN macro here, as it is not quite
      right when only one argument is floating point.  */
-- 
1.7.6.5