Building "special" binary bundles (i.e. not using guix pack)

[Thompson, David]

Hello all,

First, context and parameters:

I have a project for which I'd like to release pre-built binaries that
will "just work" on major FHS-based distributions such as Ubuntu and
Fedora.  As far as native executables go, this bundle requires Guile
and roughly a dozen C shared libraries.  The goal is to compile guile
and the additional libraries in such a way that there is *no* runpath
information embedded in the ELF files.  The goal is *not* to ship a
complete graph, down to glibc, like 'guix pack' would do.  Instead,
I'm taking the more traditional approach of bundling only the
libraries that I cannot assume will be on the user's system (say,
libvorbis) and assuming that others (like libx11) will be there.
Everything included in the bundle needs to be built against a glibc
version that is the most compatible across other distros right now.
I believe that version is 2.27.

Sure, I could just grab a bunch of pre-built stuff from Debian's
repositories and stuff it in a tarball, but I want to use Guix in
order to have a reproducible toolchain and because this bundle will
include several Guile libraries that would be a pain to build without
Guix.  I want to be able to say "Here, use this convenient binary
bundle.  If you're curious what's   Furthermore, I really want to see
Guix fill this particular niche because it could lead to an
Appimage-like feature in Guix itself.

Now, onto my issues.  I have experimented with a few different
approaches for building the bundle but haven't had success with any:

1) Using Guix packages as-is and using patchelf to strip runpath
information.  This honestly came very close to working for me, and I
think I could probably make it work with a little more debugging, but
it's very hacky and unsatisfying.  The biggest issue is that because
Guix builds everything with glibc 2.29 these days, I have to bundle
glibc 2.29, *including the dynamic linker* so I can actually run
Guile.  I sent a bundle to a friend to test and it didn't work, but
they couldn't tell me what the backtrace was because their system
locales were incompatible and thus Guile couldn't failed to print a
backtrace.  I could ship glibc 2.29 locales in the bundle, too, but
that was the last straw for me.  How much more stuff would I have to
shove into this bundle before it worked reliably? When the easy but
hacky solution starts to lose the easy part, it's time to move on.

2) Use a special GCC toolchain sans ld-wrapper and with a few other
tweaks to keep runpaths away.  This seemed promising but, because
there's no runpaths, configure scripts would fail. Although the
scripts could compile test programs, running those test programs
failed because they were unable to find libraries.  Setting
LD_LIBRARY_PATH doesn't work because then the configure script doesn't
work at all because now binaries built against glibc 2.29 are trying
to link against libraries built against 2.27 and everything explodes.
I was able to hack around this for a little while by passing
"LDFLAGS=-static-libgcc" to configure scripts, but eventually I ran
into a configure script that linked against more than libgcc in its
test programs (if you're curious, libvorbis builds a program linked
against libogg when running ./configure) and I was out of luck.  When
autoconf-generated configure scripts cannot run test executables, they
say something to effect of "are you cross-compiling? use the --host
flag".  So, I got to thinking: Should I reframe this as a
cross-compilation?  It certainly seems like I'm dealing with a special
target system.  Which leads me to...

3) Performing a cross-build.  It seems a little weird to cross-compile
for x86_64 on a machine that is natively x86_64, but I suppose it
makes sense because I'm targeting a foreign type of system.  However,
I was unable to get this to work because Guix was trying to build tons
of stuff from source, from gcc to Linux headers.  I assumed I was
doing something wrong and aborted this process.  I'm not so good at
navigating the cross-compilation code in Guix right now.

So, now I'd like to ask: Does anyone which approach I *should* be
taking here?  Does anyone have any experience they can offer?  Carl
Dong shared his approach to building bitcoin using Guix with me (see:
https://github.com/dongcarl/bitcoin-guix/blob/master/packages.scm)
which enabled me to try out approach #2 and #3, but it doesn't seem
like an exact fit, though I could just be misunderstanding something.
My hunch is that #3 is the correct approach, but it's unclear how I
setup the proper cross-compilation environment with my special gcc
toolchain.

I hope I've explained myself somewhat clearly.  Any guidance
appreciated!  If you've read this far, thank you!

- Dave