Re: Reproducing a Python project environment (using guix inferiors)

[Brandon Ellington]

Thanks again for the reply simon!

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Mon, 30 Nov 2020 at 14:39, Brandon Ellington <branjam4@gmail.com> wrote:
>
>>>> | package of interest | guix commit  | status |
>>>> |---------------------+--------------+--------|
>>>> | python-matplotlib   | "7e06086522" | bad    |
>>>> | python-pandas       | ce2cfcabfc   | bad    |
>>>> | python-networkx     | 269f100330   | good   |
>>>> | python-numpy        | 4d6ed794dd   | bad    |
>>>> | python-scipy        | 02ddafef55   | good   |
>
> Which version of <package of interest> do you need?
You can kind of get it from [2] in my initial email, but here is a
cleaner table:

| package of interest | version (rec. minimum) |
|---------------------+------------------------|
| python              |                  3.6.5 | do not rebuild
|---------------------+------------------------|
| python-matplotlib   |                  2.0.2 | used to exist in guix
| python-pandas       |                 0.22.0 |
| python-networkx     |                    2.1 |
| python-numpy        |                 1.13.3 |
| python-scipy        |                  1.0.1 |
|---------------------+------------------------|
| python-pygraphviz   |                    1.3 | bump from current guix
| python-nltk         |                  3.2.5 |
|---------------------+------------------------|
| python-wikipedia    |                  1.4.0 | importing
| python-toposort     |                    1.5 |
| python-community    |                   0.10 |
| python-louvain      |                   0.14 |
|---------------------+------------------------|
| (author's own)      |       (not applicable) |

> Because, I think that “bad” commits 7e06086522 (Jan 2018), ce2cfcabfc
> (Feb 2018) and 4d6ed794dd (Jan 2018) simply pre-date the introduction of
> «Inferior».  Therefore, they are not reachable by this mechanism.
> That’s one of the motivations for the channel “guix-past”. :-)
> 
> [...]
> 
> For commits before ~July 2018, the strategy is the one of ‘guix-past’
> channel; which I roughly described above.
>
> 1: 
> https://guix.gnu.org/en/blog/2018/multi-dimensional-transactions-and-rollbacks-oh-my/

Ah, lesson learned! I'll direct my attention toward guix-past then :)

> For commits after the complete introduction of Inferiors mechanism
> (~July 2018, please read [1]), it is ok.  Be aware that each inferior
> has a cost and if your package foo depends on X packages coming from
> inferiors, then it means you are running under the hood X time: ‘guix
> pull’ then ’guix build <dependency>’ which could require to also build
> other dependencies, therefore installing foo could be long.

Indeed I noticed this. Building the two good channels was not too bad,
but I have not tried building networkx or scipy yet. As for your
hypothetical...

> Explicitly, imagine that python-foo depends on python, python-bar and
> python-baz.  Imagine that you use 2 inferiors, one for python-bar and
> one for python-baz.  Now imagine that python-bar and python-baz depends
> both on the package python.  Then, there is a high probability to build
> 3 times tiny variants of the package python.  And maybe the 3 packages
> python-foo, python-bar and python-baz perfectly works with the same
> variant of python.  And it is worse because that applies to all the
> implicit dependencies (compilers and so on).

...I can imagine it, and that would be terrible yes. I thought python
wouldn't be a dependency of python packages in part for that reason
though. Or maybe I read this for something else like emacs and
extrapolated to python. Maybe I'm dreaming :P. In any case, the overall
idea makes sense, especially the implicit dependencies thing. For me, if
I went for full reproducibility including the older version of python,
then all these packages would work on 3.6.5, and I'd still have a lot to
learn about the implicit dependencies.

> What I will do is: a) fix a Guix version b) create a channel containing
> all the necessary variants (backporting dependencies if required, i.e.,
> copy/pasting old package definition and fix them if they does not build
> anymore) to build my short list of <packages of interest>.
>
> This is more or less the strategy used to feed the channel “guix-past“.
>
> And I will not use the inferior mechanism because it adds a lot of
> complexity and will not solve your problem since old packages are not
> reachable and/or your need to add old dependencies.

Got it. The last part of the last sentence is surprising (because the
mechanism must have been the right choice to someone/for something) but
everything else adds up.

> The time-machine is orthogonal with the way to distribute, IMHO.  It is
> simply a easy CLI to fix the Guix version.
>
> Usually, I do:
>
>   guix describe -f channels > channels.scm
>
>   edit my-manifest.scm # containing <package-of-interest>
>   
>   [.. hack my custom variants ..]
>   [.. backport variant dependencies when I need for these variants ..]
>   guix time-machine -C channels.scm
>        -- build -L /path/to/variants/ -m my-manifest.scm
>   [.. loop hack until it works ..]
>
> and then I create what my audience expects, e.g., Docker image:
>
>   guix time-machine -C channels.scm
>        -- pack -f docker -L /path/to/variants/ -m my-manifest.scm
>
> or relocatable tarball (pack -RR) or ‘system docker-image’ or whatever.
>
>
> The /path/to/variants is a Git repo and I add the files channels.scm and
> manifest.scm.  And now, it is easy to rebuild everything in the future.

Okay, definitely worth a try!

> Now, these questions and the other ones about ’guix-past’ are answered,
> right?

Yes, I appreciate you helping me out here! Now I can go down the correct
rabbit hole to build my project :)

--Brandon