Re: How should I install non-guix software?

[Chris Marusich]

Hi Luther,

Those are really good questions.  I'll try to explain what I know, and
hopefully it'll be useful to you.

Luther Thompson <address@hidden> writes:

> I don't know if this has been discussed before, but is there a best
> practice for installing software from outside the Guix package system?

For custom packages, you can define your own via the GUIX_PACKAGE_PATH.
See "Package Modules" in the Guix manual for how to do that [1].

For software installed via other package management tools like npm,
Maven, or pip, it's trickier.  Those package managers ought to work out
of the box.  I use Maven (installed via Guix) frequently on a GuixSD
system, and it's always worked well for me.

However, because (1) those package managers don't provide the same kind
of strong guarantees about dependencies that Guix does, and (2) because
GuixSD doesn't follow the FHS (for good reasons), there is definitely
the risk that some software you install via those other package managers
won't work correctly out of the box on GuixSD.

> If I have a source tarball for a program that doesn't have a Guix
> package, or if I write my own program, there just isn't any good place
> to put the installed files. Scripts have the additional problem of not
> being able to use their shebang line.

Yes, the hard-coded references are a common problem.  It's always been a
problem when software assumes that the components they require will
exist at certain absolute paths, since even among systems that don't use
Guix these paths can vary.  Software that has been designed from the
start to be portable will sometimes mitigate this problem by providing a
way for users to tell the program where its dependencies live (e.g., via
an option to the ./configure script).  However, this is still a common
problem.

I know of at least a few ways to deal with this problem.  They are:

1) Work with upstream to provide a way for users to specify the
   location.  For example, Pierre is doing this for magit right now [2].

2) Replace every hard-coded occurrence of the path with a correct path.
   We do this all the time.  For just one example, refer to the use of
   the substitute* procedure in the package definition for pies (run
   "guix edit pies").

3) Run the software in an environment where its expectations are met.

For (3), like you suggested, one way to do it is to create symlinks in
your system.  That's fine for a quick, local solution, but it isn't
appropriate for software that will be included in Guix proper.

I've sometimes created "/bin/bash" and "/usr/bin/env" on my system for
reasons like this.  When using GuixSD, you can define these kinds of
files in your OS configuration using the special-files-service-type.
See "Base Services" in the manual for details [3].

> As a more unique example, I was recently using a React tutorial. IIRC,
> I ran the command `npm start`. It tried to run another installed script.
> I had to manually change the shebang line of that other script to point
> to /home/luther/.guix-profile/bin/node to get it to run.

The build systems of Guix (e.g., gnu-build-system) generally have a
build phase that automatically finds and patches shebang lines,
replacing them with absolute paths to the corresponding interpreters in
the store.  Unfortunately, if other package managers like npm, Maven,
pip, etc. do not provide a way to change hard-coded paths - and I don't
think any of them do - this will always be a problem.  In my opinion, if
you have to use these kinds of tools to install software on GuixSD, the
best compromise is probably to create symlinks in your system as above.

As far as I know, there isn't a good, holistic way in GuixSD to deal
with hard-coded paths in software that has been installed via other
package managers like npm.  If someone knows better, I'd love to hear
otherwise!

> 1. Write a Guix package for every little piece of software we want to
> run. I have yet to successfully do this, but I might decide to learn
> someday. I already know Scheme. AFAIK, this won't work for node
> packages, but it should work for everything else. This solution would
> be a lot of work for users.

In some ways, this is the best approach.  However, I know it's easier
said than done.  If you're ever feeling stuck, you can ask for help here
as long as the software you're trying to package is free software.  It's
also helpful to look at the thousands of existing examples in the Guix
source tree, which are located in the gnu/packages directory.

> 2. Put ~/bin in our $PATH and install all our executables there. This
> wouldn't work for the majority of software that installs more than just
> executable files.

I'm not sure what you mean here.  How would putting ~/bin in your $PATH
enable software that hard-codes a path of "/bin/bash" to find the bash
program?

> 3. Instead of installing the software, run it directly from whatever
> directory we unpacked/compiled it to.

Wouldn't this also fail to address the problem of hard-coded paths?

> 1. Make a soft link at /usr/bin/env in the root filesystem pointing
> to /run/current-system/profile/bin/env. This seems like the simplest
> solution, but I figure there must be some reason the devs haven't
> already done this.

Yeah, I think creating symlinks like this is probably the easiest way to
get "foreign" software working on a GuixSD system.  It doesn't always
work, but when it does, it's an easy hack.

However, like I said, that solution isn't appropriate for software that
will be included in Guix proper.  This is because it just isn't
necessary: if the software is packaged in Guix proper, then all of its
dependencies are guaranteed to exist (at exactly the intended versions)
in the store.

In fact, if we added symlinks like "/bin/bash" to GuixSD by default, it
would mask certain kinds of packaging problems.  If I packaged a piece
of software that uses the hard-coded path /bin/bash at runtime, it would
run fine on my system because I have a "/bin/bash" symlink, which points
to some (indeterminate) version of bash.  When you install the same
package, your /bin/bash symlink might point to a different version of
bash, so the software might behave differently for you than it does for
me.  In addition, if for example you make a bundle of the software using
"guix pack" and ship it to a system that doesn't have /bin/bash, the
software might break entirely.

My understanding is that one of the reasons why Guix doesn't follow the
FHS is to make it difficult to accidentally package software that will
break in these sorts of ways.  If /bin/bash is missing from my system
when I package software that assumes it will exist at that path, I will
be more likely to notice that the software depends on something outside
the store.  I can then fix it by replacing the path with an exact
version of bash that is managed by Guix.  In this way, the problems
mentioned above will not happen, since whenever somebody installs the
software via Guix, it is guaranteed that the installed software will use
the exact same version of bash that I specified in the package
definition (assuming that the user isn't using a different version of
Guix)

In this way, GuixSD is designed to be a "pure" system that tries hard to
prevent undeclared dependencies from creeping into your environment.
Undeclared dependencies are bad because they can interfere with or break
software.  The usual FHS runs counter to this goal, since it encourages
software to make specific assumptions about the environment.  With Guix
(and Nix), it is no longer necessary to make those kinds of assumptions
in the first place, since all the dependencies are always known exactly.

> 2. Know which interpreter we need for each program and invoke that
> interpreter explicitly.

Yes, this is also an option, but as you might imagine, it doesn't scale
past a handful of scripts/programs.

> 3. Manually change the shebang line for every script to point into
> our .guix-profile. I don't like this idea, because it would mean
> hard-coding the name of our home directory into every script. We can't
> really do that with programs that we want to hack on and publish.

Yeah, I agree that isn't a great solution, either.  But it will work in
a pinch, too.

> Currently, I invoke binaries from their own directories and for
> scripts, I invoke the interpreters. I've only been using GuixSD for a
> short time, so I doubt that this is a fool-proof solution. How do
> you guys handle this?

In sum, I guess the most common things that I do are (in no particular
order):

* Create symlinks in my system when needed.

* Manually patch specific files (e.g., replacing their shebang lines, or
  using patchelf to replace the ld interpreter path).

* Create package definitions either in my GUIX_PACKAGE_PATH or in a
  special guix.scm file (which you can install into a custom profile
  using "guix package -f my-file.scm").  Ultimately, if it works out, I
  upstream my packages to Guix proper so everyone can benefit.

Finally, an alternative of last resort is to run the problematic
software in a VM using a more traditional GNU/Linux distribution.
However, I didn't mention that until now, since it's basically
equivalent to giving up on trying to run it in GuixSD itself.

Anyway, I hope that helps!  Thanks for bearing with my very long reply.

Footnotes: 
[1]  
https://www.gnu.org/software/guix/manual/en/html_node/Package-Modules.html#Package-Modules

[2]  https://lists.gnu.org/archive/html/help-guix/2018-08/msg00032.html

[3]  
https://www.gnu.org/software/guix/manual/en/html_node/Base-Services.html#index-special_002dfiles_002dservice_002dtype

-- 
Chris

signature.asc
Description: PGP signature