physmem: Port better to Linux

[Bruno Haible]

I'm writing a unit test that takes 12 GiB of RAM, and want to test whether it
is adequate to run this test.

In a first attempt, I wrote:

    if (physmem_total () / INT_MAX < 6.0)
      /* run the test */

but this resulted in the test not being run, although my machine has plenty
of RAM:

$ ./test-vasnprintf-big 
avail = 4534.64 MiB
Skipping test: not enough memory available
$ free
               total        used        free      shared  buff/cache   available
Mem:           63579       22159        4533        1184       36887       39533
Swap:          19072        2566       16506

The reason is that physmem_available(), like sysconf (_SC_AVPHYS_PAGES), 
returns,
as documented in the glibc manual [1]
  "the amount of memory the application can use without hindering any other
   process (given that no other process increases its memory usage)"
and that is the number of "free" pages.

In my case, with 4.5 GiB "free" pages plus 39 GiB "available" pages, it is
completely adequate to run a test that takes 12 GiB of RAM. The Linux
memory management usually keeps the number of free pages relatively small
over time (because free memory is wasted memory). So, to determine
whether it is OK to take a certain amount of memory — of course, while
slowing down other processes a bit, but without crashing the system — we
must look at the other categories of memory. The figures from /proc/meminfo
are quite detailed:

$ cat /proc/meminfo 
MemTotal:       65105348 kB
MemFree:         4488440 kB
MemAvailable:   40354724 kB
Buffers:         2738416 kB
Cached:         30930548 kB
SwapCached:       239900 kB
Active:         15246788 kB
Inactive:       39476552 kB
Active(anon):    1143740 kB
Inactive(anon): 21123360 kB
Active(file):   14103048 kB
Inactive(file): 18353192 kB
Unevictable:         208 kB
Mlocked:             208 kB
SwapTotal:      19530748 kB
SwapFree:       16902908 kB
Dirty:               100 kB
Writeback:             0 kB
AnonPages:      20842972 kB
Mapped:          1855324 kB
Shmem:           1212724 kB
KReclaimable:    4129976 kB
Slab:            4644636 kB
SReclaimable:    4129976 kB
SUnreclaim:       514660 kB
KernelStack:       64720 kB
PageTables:       165636 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    52083420 kB
Committed_AS:   49274636 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      116336 kB
VmallocChunk:          0 kB
Percpu:            23040 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:    21034368 kB
DirectMap2M:    36941824 kB
DirectMap1G:     9437184 kB

These figures are documented in [2][3][4][5]. In particular, the
"Inactive(file)" number is memory that is used in the file cache.
When pages from this pool are reclaimed, of course some other processes
will slow down. But we know that reclaiming less than 100% of the
pages from the file cache will not bring the machine to its knees.

So, here is a patch that adds another function physmem_claimable,
which returns a larger value than physmem_available. A call
  physmem_claimable (0.5)
returns 4 GiB + 0.5 * 18 GiB = 13 GiB, which is sufficient for
running the 12 GiB test.

[1] 
https://www.gnu.org/software/libc/manual/html_node/Query-Memory-Parameters.html
[2] 
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/s2-proc-meminfo
[3] https://man7.org/linux/man-pages/man5/proc.5.html
[4] 
https://superuser.com/questions/521551/cat-proc-meminfo-what-do-all-those-numbers-mean
[5] https://www.baeldung.com/linux/proc-meminfo


2024-04-24  Bruno Haible  <bruno@clisp.org>

        physmem: Port better to Linux.
        * lib/physmem.h (physmem_total, physmem_available): Add documentation.
        (physmem_claimable): New declaration.
        * lib/physmem.c: Include <fcntl.h>, <stdio.h>, full-read.h.
        (get_meminfo): New function.
        (physmem_claimable): Renamed from physmem_available. Add logic for
        aggressivity > 0.
        (physmem_available): New function.
        * modules/physmem (Depends-on): Add full-read.

diff --git a/lib/physmem.h b/lib/physmem.h
index 957e629180..542dbf214c 100644
--- a/lib/physmem.h
+++ b/lib/physmem.h
@@ -25,9 +25,27 @@ extern "C" {
 #endif


+/* Returns the total amount of physical memory.
+   This value is more or less a hard limit for the working set.  */
 double physmem_total (void);
+
+/* Returns the amount of physical memory available.
+   This value is the amount of memory the application can use without hindering
+   any other process (assuming that no other process increases its memory
+   usage).  */
 double physmem_available (void);

+/* Returns the amount of physical memory that can be claimed, with a given
+   aggressivity.
+   For AGGRESSIVITY == 0.0, the result is like physmem_available (): the amount
+   of memory the application can use without hindering any other process.
+   For AGGRESSIVITY == 1,0, the result is the amount of memory the application
+   can use, while causing memory shortage to other processes, but without
+   bringing the machine into an out-of-memory state.
+   Values in between, for example AGGRESSIVITY == 0.5, are a reasonable middle
+   ground.  */
+double physmem_claimable (double aggressivity);
+

 #ifdef __cplusplus
 }
diff --git a/lib/physmem.c b/lib/physmem.c
index e6eb26b5f3..5c226b8abf 100644
--- a/lib/physmem.c
+++ b/lib/physmem.c
@@ -22,25 +22,28 @@
 
 #include "physmem.h"
 
+#include <fcntl.h>
+#include <stdio.h>
 #include <unistd.h>
 
-#if HAVE_SYS_PSTAT_H
+#if HAVE_SYS_PSTAT_H /* HP-UX */
 # include <sys/pstat.h>
 #endif
 
-#if HAVE_SYS_SYSMP_H
+#if HAVE_SYS_SYSMP_H /* IRIX */
 # include <sys/sysmp.h>
 #endif
 
 #if HAVE_SYS_SYSINFO_H
+/* Linux, AIX, HP-UX, IRIX, OSF/1, Solaris, Cygwin, Android */
 # include <sys/sysinfo.h>
 #endif
 
-#if HAVE_MACHINE_HAL_SYSINFO_H
+#if HAVE_MACHINE_HAL_SYSINFO_H /* OSF/1 */
 # include <machine/hal_sysinfo.h>
 #endif
 
-#if HAVE_SYS_TABLE_H
+#if HAVE_SYS_TABLE_H /* OSF/1 */
 # include <sys/table.h>
 #endif
 
@@ -51,13 +54,16 @@
 #endif
 
 #if HAVE_SYS_SYSCTL_H && !(defined __GLIBC__ && defined __linux__)
+/* Linux/musl, macOS, *BSD, IRIX, Minix */
 # include <sys/sysctl.h>
 #endif
 
-#if HAVE_SYS_SYSTEMCFG_H
+#if HAVE_SYS_SYSTEMCFG_H /* AIX */
 # include <sys/systemcfg.h>
 #endif
 
+#include "full-read.h"
+
 #ifdef _WIN32
 
 # define WIN32_LEAN_AND_MEAN
@@ -203,11 +209,84 @@ physmem_total (void)
   return 64 * 1024 * 1024;
 }
 
-/* Return the amount of physical memory available.  */
+#if defined __linux__
+
+/* Get the amount of free memory and of inactive file cache memory, and
+   return 0.  Upon failure, return -1.  */
+static int
+get_meminfo (unsigned long long *mem_free_p,
+             unsigned long long *mem_inactive_file_p)
+{
+  /* While the sysinfo() system call returns mem_total, mem_free, and a few
+     other numbers, the only way to get mem_inactive_file is by reading
+     /proc/meminfo.  */
+  int fd = open ("/proc/meminfo", O_RDONLY);
+  if (fd >= 0)
+    {
+      char buf[4096];
+      size_t buf_size = full_read (fd, buf, sizeof (buf));
+      close (fd);
+      if (buf_size > 0)
+        {
+          char *buf_end = buf + buf_size;
+          unsigned long long mem_free = 0;
+          unsigned long long mem_inactive_file = 0;
+
+          /* Iterate through the lines.  */
+          char *line = buf;
+          for (;;)
+            {
+              char *p;
+              for (p = line; p < buf_end; p++)
+                if (*p == '\n')
+                  break;
+              if (p == buf_end)
+                break;
+              *p = '\0';
+              if (sscanf (line, "MemFree: %llu kB", &mem_free) == 1)
+                {
+                  mem_free *= 1024;
+                }
+              if (sscanf (line, "Inactive(file): %llu kB", &mem_inactive_file) 
== 1)
+                {
+                  mem_inactive_file *= 1024;
+                }
+              line = p + 1;
+            }
+          if (mem_free > 0 && mem_inactive_file > 0)
+            {
+              *mem_free_p = mem_free;
+              *mem_inactive_file_p = mem_inactive_file;
+              return 0;
+            }
+        }
+    }
+  return -1;
+}
+
+#endif
+
+/* Return the amount of physical memory that can be claimed, with a given
+   aggressivity.  */
 double
-physmem_available (void)
+physmem_claimable (double aggressivity)
 {
 #if defined _SC_AVPHYS_PAGES && defined _SC_PAGESIZE
+# if defined __linux__
+  /* On Linux, sysconf (_SC_AVPHYS_PAGES) returns the amount of "free" memory.
+     The Linux memory management system attempts to keep only a small amount
+     of memory (something like 5% to 10%) as free, because memory is better
+     used in the file cache.
+     We compute the "claimable" memory as
+       (free memory) + aggressivity * (inactive memory in the file cache).  */
+  if (aggressivity > 0.0)
+    {
+      unsigned long long mem_free;
+      unsigned long long mem_inactive_file;
+      if (get_meminfo (&mem_free, &mem_inactive_file) == 0)
+        return (double) mem_free + aggressivity * (double) mem_inactive_file;
+    }
+# endif
   { /* This works on linux-gnu, kfreebsd-gnu, solaris2, and cygwin.  */
     double pages = sysconf (_SC_AVPHYS_PAGES);
     double pagesize = sysconf (_SC_PAGESIZE);
@@ -312,6 +391,12 @@ physmem_available (void)
   return physmem_total () / 4;
 }
 
+/* Return the amount of physical memory available.  */
+double
+physmem_available (void)
+{
+  return physmem_claimable (0.0);
+}
 
 #if DEBUG
 
diff --git a/modules/physmem b/modules/physmem
index 2cb7e7f64a..e1a4a76700 100644
--- a/modules/physmem
+++ b/modules/physmem
@@ -8,6 +8,7 @@ m4/physmem.m4
 
 Depends-on:
 unistd
+full-read
 
 configure.ac:
 gl_PHYSMEM