Re: guix gc: smarter collection & guix.el manual deletion

[Chris Marusich]

Pierre Neidhardt <address@hidden> writes:

> Thanks for mentioning this.  I'm very confused by the documentation however:
>
> ‘--gc-keep-outputs[=yes|no]’
>      Tell whether the garbage collector (GC) must keep outputs of live
>      derivations.
>
>      When set to “yes”, the GC will keep the outputs of any live
>      derivation available in the store—the ‘.drv’ files.  The default is
>      “no”, meaning that derivation outputs are kept only if they are GC
>      roots.  *Note Invoking guix gc::, for more on GC roots.
>
> Specifically this: "derivation outputs are kept only if they are GC roots'".
> Would that mean that other live outputs are deleted?!?  I must get this 
> wrong! :p

Live outputs are never eligible for garbage collection.  The behavior of
these options might be a little non-obvious, so I'll explain what I
understand about it, and hopefully it will help.

First, let's assume that you've set both --gc-keep-derivations=no and
--gc-keep-outputs=no.  Let's also suppose that you've installed
address@hidden into your profile.  Since your profile is a GC root, this
means that address@hidden's output (e.g., /gnu/store/...krfy-hello-2.10) is
live.  However, its derivation isn't live, since the derivation is
neither in your profile nor in the output's closure.

To query for the deriver of address@hidden, you can do this:

--8<---------------cut here---------------start------------->8---
$ guix build -d address@hidden
/gnu/store/p2hmc4wv59kxvhvxa1dwjjps0g38ikc1-hello-2.10.drv
--8<---------------cut here---------------end--------------->8---

That derivation describes precisely how to build address@hidden, all the way
down to how to fetch the sources.  That's because it refers to other
dependencies.  Here they are:

--8<---------------cut here---------------start------------->8---
$ guix gc --references $(guix build -d address@hidden)
/gnu/store/1bjlh3cyij46a3mwi0ikyrn1pb70bd8m-gawk-4.2.1.drv
/gnu/store/1dv1gyyqfn50i2zrq4fvr7fbgwm3hlkx-hello-2.10-guile-builder
/gnu/store/2n4fdhjq47cgifbp2wsmyapqxih73bm4-ld-wrapper-0.drv
[... in total, there are 25 store items like this ...]
--8<---------------cut here---------------end--------------->8---

You can consider these store items to be the "direct dependencies" of
the address@hidden derivation.  Many of them are derivations themselves,
which makes sense because you cannot start the address@hidden derivation
until after all the things it requires have been built.  Furthermore,
these derivations may in turn refer to other derivations.  Here are the
"direct and transitive dependencies" of the address@hidden derivation (which
includes the address@hidden derivation itself):

--8<---------------cut here---------------start------------->8---
$ guix gc --requisites $(guix build -d address@hidden)
/gnu/store/xc0wliwjngcsx26wh65dx07xnas4v146-gcc-4.9.4.tar.xz-builder
/gnu/store/lmfvf4iwkzn4wibvb7ik3cadxq07pjvi-gcc-libvtv-runpath.patch
/gnu/store/d57l8pc992bdqd20l5piqk9k47dqirzm-gcc-4.9-libsanitizer-fix.patch
/gnu/store/5752c05gravrwk2kvy1n2zrgl9j0kxhg-gcc-4.9.4.tar.bz2.drv
[... in total, there are 308 store items like this ...]
--8<---------------cut here---------------end--------------->8---

These 308 store items make up the closure of the address@hidden derivation.
Compare this to the closure of address@hidden's output:

--8<---------------cut here---------------start------------->8---
$ guix gc --requisites $(guix build address@hidden)
/gnu/store/1r3dlhi2vasb8yw630728jlrk40mygj1-bash-static-4.4.19
/gnu/store/vla5j7pbkpcp39lsdfsmz7m9azn48lr4-gcc-5.5.0-lib
/gnu/store/l4lr0f5cjd0nbsaaf8b5dmcw1a1yypr3-glibc-2.27
/gnu/store/bihfrh609gkxb9dp7n96wlpigiv3krfy-hello-2.10
--8<---------------cut here---------------end--------------->8---

The output's closure contains only 4 items!  And if you examine these 4
items, you'll find that none of them appear in the closure of the
address@hidden derivation.  Therefore, even though the output of address@hidden
is live because you've installed it into your profile, none of those 309
store items that were required for building address@hidden are live.  They
are all eligible for garbage collection.

However, if you set --gc-keep-derivations=yes, then the address@hidden
derivation will be treated as live.  As a result, every store item in
the address@hidden derivation's closure becomes live, also.  In this way,
--gc-keep-derivations=yes causes liveness to flow from outputs to
derivations.

But we're not done yet.  Consider one of the derivations that shows up
in the address@hidden' derivation's closure:

/gnu/store/5752c05gravrwk2kvy1n2zrgl9j0kxhg-gcc-4.9.4.tar.bz2.drv

This derivation builds a tarball of gcc's sources.  Let's build it:

--8<---------------cut here---------------start------------->8---
$ guix build /gnu/store/5752c05gravrwk2kvy1n2zrgl9j0kxhg-gcc-4.9.4.tar.bz2.drv
/gnu/store/1j3mqrcp3y4xlb9jl5d0ri5aszn8mfii-gcc-4.9.4.tar.bz2
--8<---------------cut here---------------end--------------->8---

The output path of this derivation is
/gnu/store/...mfii-gcc-4.9.4.tar.bz2, which appears in neither the
closure of address@hidden's derivation nor the closure of address@hidden's
output.  So, if you had built this derivation while building address@hidden,
the derivation's output (/gnu/store/...mfii-gcc-4.9.4.tar.bz2) would be
eligible for garbage collection even if you had specified
--gc-keep-derivations=yes.

This is where --gc-keep-outputs=yes comes in.  If you specify it, then
the /gnu/store/...mfii-gcc-4.9.4.tar.bz2 store item will be treated as
live, since it is the output of a live derivation
(/gnu/store/...kxhg-gcc-4.9.4.tar.bz2.drv).  In this way,
--gc-keep-outputs=yes causes liveness to flow from derivations to
outputs.

In Guix, the defaults are --gc-keep-outputs=no and
--gc-keep-derivations=yes.  In this configuration, derivations tend to
be kept, and outputs tend to be collected.  Derivations usually don't
take up much space (invoke "guix size $(guix build -d hello)" to get an
idea of how small they are), but their outputs can be large.  If you set
--gc-keep-outputs=yes in addition to --gc-keep-derivations=yes, you will
retain package outputs that you install, as well as all the derivations
necessary for building the package, as well as all the outputs that you
built while running those derivations.  If you build from source instead
of downloading substitutes, this means you'll have a lot of intermediary
outputs stored in your local file system, and you won't be able to
collect them, since they'll all be live.

You should try setting both to yes explicitly and see if this makes your
experience any better.  Hopefully it does!  However, there is a chance
it might actually cause even more disk usage, and you might not be able
to collect enough garbage to free up space.

Anyway, I hope that helps clarify what these options do.  I had
forgotten about it, myself, so it was fun to remind myself how it works.

Pierre Neidhardt <address@hidden> writes:

> The store does not have timestamps, so removing old items would not work
> I guess.  But what about removing older versions first?  Say I have
> glibc-2.24 and glibc-2.25, only remove glibc-2.24 if that's enough to
> free the required space.
>
> Maybe we could introduce a "whitelist" of packages to delete last.

These might help, but I think the options I suggested are the best thing
to try first.  Do they help in your case?  You might need to
aggressively delete old profile generations, also, to make sure you have
some garbage to collect.

> In the end I figured that the most convenient way to clean up the
>store
> might be to do it manually with "guix gc --delete PATH".  Well, not so
> from the commandline, but with guix.el I thought we could do something
> nice.  So I gave it a (quick&dirty™) shot:
>
>       https://gitlab.com/emacs-guix/emacs-guix/issues/2
>
> With guix.el loaded, `eval' my example and run "M-x guix-store".
> This will return a list of all the dead links, which you can sort by
> name or by size.  You can then mark items ("m") and delete them ("k").
>
> What do you people think?

I don't use emacs-guix, so I can't comment on it, I'm afraid.  Maybe
someday I'll get around to using it, and then I can say something
useful!

> While I'm at it, I'd like to note that something might be wrong with the
> `-F` option: I never get the promised amount of free space back, only
> about 2/3 of it.

I think that's normal.  My understanding is that if you say "guix gc -F
3GiB", you aren't asking Guix to free 3 GiB; you're asking Guix to try
to free enough space so that when it's done, the store will have 3 GiB
of free space.  For example, if the store begins with 2 GiB of free
space and there is 5 GiB of garbage, Guix will only collect 1 GiB in
order to bring the free space up to 3 GiB.


-- 
Chris

signature.asc
Description: PGP signature