the page size bug

[Andy Wingo]

Hi,

Guile 2.2 uses ELF to format its object files.  ELF objects are composed
of *sections* and *segments*.  Sections are things like .data, .strtab,
and so on; segments contain sections with similar permissions
(read-only, read-write, etc).  See
https://www.gnu.org/software/guile/docs/master/guile.html/Object-File-Format.html
for more details.

Parts of different sections refer to each other.  For example the
read-only .rtl-text code section refers to mutable values in the .data
section, by relative offset.  It's the linker's job to make all of these
connections, and the loader's job to ensure they are valid when the file
is loaded.

The loader has two paths: an mmap path and a malloc path.  The malloc
path just slurps the file into memory; the relative references in the
file correspond to relative references in memory.  With the malloc path,
everything is mutable from the operating system's point of view.

The problem with the malloc path is that not only is it a bit slower, as
it has to page in data that might not be needed, it doesn't share memory
between processes.  So if the system supports mmap, Guile will use mmap
to load its ELF images (.go files).

The mmap path likewise just mmaps the file into memory.  The operating
system will lazily page in data from disk as needed.  Writable segments
ends up being process-local, but read-only segments can share memory
between different processes.  This lowers Guile's resident memory
footprint, especially when many Guile processes are live.

What ends up happening in the mmap case is that the whole file is mapped
into memory, then the writable segments are made writable via
mprotect().  This only works if the writable segments are page-aligned.

And with that prelude out of the way, here's the bug: Guile currently
assumes that 4096 is a multiple of the page size.  So the mprotect fails
to make the segment writable, and runtime relocations that mutate the
segment cause segmentation faults.

One solution to this issue would be to choose target-specific page
sizes.  This is still a little tricky; on amd64 for example, systems
commonly have 4KB pages, but they are allowed by the ABI to have any
multiple-of-2 page size up to 64 KB.  On Cygwin, pages are 4kB but they
can only be allocated 16 at a time.  MIPS and ARM64 can use 64K pages
too and that's not uncommon.

At the current time, in Guile we would like to reduce the number of
binaries we ship to the existing 32-or-64-bit and big-or-little-endian
variants, if possible.  It would seem that with the
least-common-multiple of 64 KB pages, we can do that.

See https://github.com/golang/go/issues/10180 for a discussion of this
issue in the Go context.

So my proposal is to increase the page size to which we align our
segments to 64KB.  This will increase the size of our .go files, but not
the prebuilt/ part of the tarball as that part of the file will be
zeroes and compress well.  Additionally on a system with 4KB pages, the
extra padding will never be paged in, nor read from disk (though it
causes more seeking etc so on spinning metal it's a bit of a lose).

On many 64-bit platforms, binutils currently defaults to aligning
segments on 2MB boundaries.  It does so by making the file and the
memory images not the same: the pages are all together on disk, but then
when loading, the loader will mmap a region "memsz" large which might be
greater than the file size, then map segments into that region.  I would
like to avoid this complication for now.  We can consider adding it in
the future in a compatible way in 2.2 if it is important.

Thoughts welcome :)  I am going to give this a go now.

Andy