[bug#53818] Improving updaters and ‘guix refresh’

[Ludovic Courtès]

Hi,

Nicolas Goaziou <mail@nicolasgoaziou.fr> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Regarding “specifying many packages”, do examples like these work for
>> you:
>>
>>   • guix refresh -t elpa
>
> I don't find it very useful in practice. As a user, the packages I'm
> interested in probably rely on more than one updater. I'm not even
> supposed to know what updater relates to a given package.

Right, that’s more for packagers than for users.

As a user, what works better is:

  guix refresh -r $(guix package -I |cut -f1) -s non-core

… or simply ‘--with-latest’, if I’m not interested in updating package
definitions.

>>   • guix refresh $(guix package -A ^emacs- | cut -f1)
>
> This one is interesting. This illustrates that the UI is, from my point
> of view, a bit lacking. It would be a nice improvement to add a regexp
> mechanism built-in, like in "guix search".

Makes sense, we can do that.

> In any case, this fails after reporting status of around 50 packages,
> with this time:
>
>   real        0m41,881s
>   user        0m12,155s
>   sys 0m0,726s

How does it fail?  If it’s the GitHub rate limit, then there’s only one
answer: you have to provide a token.

> Assuming I don't get the "rate limit exceeded" error, at this rate, it
> would take more than 15 minutes to check all the packages in
> "emacs-xyz.scm". This is a bit long.

> I don't see how this could reasonably be made faster without relying on
> an external centralized service doing the checks regularly (e.g., once
> a day) before the user actually requests them.

Maybe you’re right, but before jumping to the conclusion, we have to
investigate a bit.  Like I wrote, the ‘gnu’ updater for instance fetches
a single file that remains in cache afterwards—the cost is constant.

We should identify updaters that have linear cost and check what can be
done.  ‘github’, ‘generic-html’, and ‘generic-git’ are of that kind.

Now, the command I gave above looks at 1,134 packages, so is it even
something you want to do as a packager?

>>   • guix refresh -r emacs-emms
>
> It also fails with the "rate limit exceeded". While this sounds
> theoretically nice, I wouldn't know how to make use of it yet.
>
>>   • guix refresh -s non-core -t generic-git
>
> See above about "-t elpa".
>
>>   • guix refresh -m packages-i-care-about.scm
>
> Yes, obviously, this is a nice, too. However, it doesn't scale if you
> need to specify 1000+ packages.

You can use ‘fold-packages’ and have three lines that return a manifest
of 10K packages if you want it.

Honestly, since I mostly rely on others these days :-), I’m no longer
sure what the packager’s workflow is.  Also, the level of coupling
varies greatly between, say, a C/C++ package and a set of
Python/Emacs/Rust packages.

I find that ‘guix refresh’ works fine for loosely-coupled C/C++ packages
where often you’d want to upgrade packages individually.

But for Python and Emacs packages, what do we want?  Do packagers always
want to check 1K+ packages at once?  Or are there other patterns?

>> If not, what kind of selection mechanism could help?  ‘-s’ currently
>> accepts only two values, but we could augment it.
>
> Besides regexp matching, it may be useful to filter packages per module,
> or source file name. Package categories is a bit awkward, tho, and
> probably not satisfying.

We can add options to make it more convenient, but it’s already
possible:

  guix refresh $(guix package -A | grep emacs-xyz.scm | cut -f1)

>> I realize this is going off-topic, but let’s see if we can improve the
>> existing infrastructure to make it more convenient.
>
> Is it really off-topic?
>
> Anyway, all of this is only one data point, and, as a reminder,
> I certainly don't want to disparage either Xinglu Chen's work, or
> current "guix refresh" functionality.

Yup, same here!

I think we have nice infrastructure but you raise important
shortcomings.  What Xinglu Chen did might in fact be one way to address
it, and there may also be purely UI issues that we could address.

Thanks,
Ludo’.