bug#65665: package-mapping with #:deep? #t doesn't get all the implicit inputs

[Ludovic Courtès]

Hello,

Ulf Herrman <striness@tilde.club> skribis:

> That and a growing thirst for a nuclear option for package rewriting
> brought about by trying to debug deep transformations while
> simultaneously experimenting with rewriting a manifest of around 270
> packages.

On that topic of a catch-all option for rewriting: there’s an unused
procedure called ‘map-derivation’, which performs rewriting at the level
of derivations.  It’s harder to use, more expensive, but who knows,
perhaps we’ll find a motivating use case…  (Allowing for graph rewriting
has been a major goal for me since the beginning.)

> There are really several distinct issues at play here:
> 1. The case of #:qtbase and #:guile being invisible to package-mapping.
>    This is what I first noticed, and cannot be fixed without modifying
>    the build systems.  This is what prompted looking for packages in
>    package and bag arguments, and recursing into lists therein (just in
>    case an argument took a list of packages).

How are #:qtbase and #:guile invisible to package mapping?

They appear in the bag inputs and thus are definitely visible to
‘package-mapping’, as a I showed with the CMake and Python examples in
this thread.

For good measure :-) here’s an example with #:qtbase:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build qgit -n
substitute: updating substitutes from 'http://192.168.1.48:8123'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
0.4 MB would be downloaded:
  /gnu/store/7b20q17yg90b62404chgbnwgvd6ry1qf-qgit-2.10
$ ./pre-inst-env guix build qgit -n --with-latest=qtbase
following redirection to 
`https://mirrors.ocf.berkeley.edu/qt/official_releases/qt/'...
following redirection to 
`https://mirrors.ocf.berkeley.edu/qt/official_releases/qt/6.6/'...
guix build: warning: cannot authenticate source of 'qtbase', version 6.6.0

Starting download of /tmp/guix-file.CTehnY
>From 
>https://mirrors.ocf.berkeley.edu/qt/official_releases/qt/6.6/6.6.0/submodules/qtbase-everywhere-src-6.6.0.tar.xz...
 …-src-6.6.0.tar.xz  46.1MiB                                                    
                                                                    12.9MiB/s 
00:04 ▕██████████████████▏ 100.0%
substitute: updating substitutes from 'http://192.168.1.48:8123'... 100.0%
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://guix.bordeaux.inria.fr'... 100.0%
The following derivations would be built:
  /gnu/store/paixxkdaakv55bffggxx4l9hiknl8i5r-qgit-2.10.drv
  /gnu/store/f9fdjk1g1s1aqmlmi4clla2kqns7283v-qtbase-6.6.0.drv
0.4 MB would be downloaded:
  /gnu/store/nl9dadzfmjm9wg7v3r31jkx773dl683x-module-import-compiled
  /gnu/store/6zryxmypw0wygayc9pvhyxkx47w0lyci-gperf-3.1
  /gnu/store/a57n7wy8mdi7l52pr4zg07132blgj5xp-qgit-2.10-checkout
--8<---------------cut here---------------end--------------->8---

Now, package transformation options are almost all implemented in terms
of ‘package-input-rewriting/spec’, and I have to admit that it’s much
easier to use than ‘package-mapping’.  The latter is low-level and it
can be a source of headaches.

> 2. The (perceived) case of packages hiding inside arguments.  In
>    hindsight, this was probably actually (3) causing this, though it's
>    hard to tell because I discovered it last.  I attempted to resolve
>    this by recursing through <gexp>s and <gexp-input>s.

Yeah.  I think we have to understand that “hiding” is to some extent
inevitable; <package> is more concise than <bag>, which is more concise
than <derivation>.  There are extra “inputs” showing up when we go from
one abstraction to the one below.

> 3. `this-package' referring to the inherited package in thunked fields,
>    rather than the package inheriting them.  This is what prompted the
>    use of package-{inputs,arguments,etc}-with-package.

Ah yes, I agree; I reported it here:

  https://issues.guix.gnu.org/50335

I think we can discuss it separately though, in that bug report
probably.

> (1) could be resolved in several ways.  I'm partial to looking for
> arguments whose values are packages and transforming them (when #:deep?
> #t is specified), and adjusting the build systems to make that work
> consistently, which is what I've done.
>
> (2) is probably not an issue, though it occurs to me that the technique
> of recursively searching through arguments looking for packages could be
> used to implement a sort of automated "transformability" check, which
> could help a lot when trying to debug transformations.

OK.

> (3) is a major issue; the entire strategy of using `this-package-input'
> to enable transformations breaks because of it.  My fix works for me at
> least, though it requires exposing additional low-level procedures and
> transformation authors using them.  I'll open another bug about it, as
> requested.

Yeah, understood.

Thanks,
Ludo’.