Re: Guix Docker image inflation

[Stephen Scheck]

On Sun, May 31, 2020 at 12:31 AM Chris Marusich <cmmarusich@gmail.com>
wrote:

> > Also, layers are helpful in the case of someone pulling down daily
> > Guix Docker images on a frequent basis, because then only the new,
> > ideally small layers need to be downloaded, whereas if you rebase for
> > every image build, you'd have to download the entire image every day.
>
> That is true, but suppose I have the following 3 images:
>
> - Image A: A base image created in January 2020.
> - Image B: Based on A, and I ran "guix pull" in February 2020.
> - Image C: Based on A, and I ran "guix pull" in June 2020.
>
> I would guess that the size difference between A and B is approximately
> the same as the difference between A and C.  It'll be different, of
> course, but generally the size difference between A and C should not
> grow linearly with time, since "guix pull" is only going to install at
> most the total closure of things necessary to build and run Guix, which
> doesn't increase much in size as time goes on.  However, when you
> daisy-chain the images every day, the image size will grow linearly with
> time because the contents of all the previous layers is carried forward.
>
> > My build script issues several `docker exec <container> <command>`
> > sequences, followed by a `docker commit <container>`. Intermediate
> > changes to the container file system prior to the commit do not
> > generate layers, only the net changes after the commit.
>
> There are two problems here.  One is that the image size grows without
> bound.  The other is that guix-daemon is failing to GC store items in
> the Docker container.  Although they are both concerning, the latter is
> not the cause of the former.
>
> If you install new store items (e.g., via "guix pull"), make them dead,
> and then GC them, all in the same container before running "docker
> commit", then I agree: those GC'd store items would not persist in a
> layer anywhere.  However, I don't think that's what's happening here.
> Sure, there might be a few store items like this, but in practice, there
> will be many store items from the previous image which began live but
> became dead when you ran "guix pull" and deleted your old profile
> generations.  It is those store items that are adding the most space to
> your image.
>

Yes, I get this. I never expected the container to stay constant in size,
but I
was hoping daily pulls would result in relatively low image growth. It's not
clear to me if any of the items which should get GC'd but don't are just
ephemeral build results, in which case growth might be tolerable with an
occasional rebase (perhaps monthly or bi-monthly).

But I'm now starting to doubt my whole approach because it seems like
there are some fundamental GC problems with running a live Guix system
inside a container.


> Besides store items, I noticed two other things about your images:
>
> - The contents of /var is growing slowly without bound, but it isn't
>   nearly as bad as the contents of /gnu/store.  This is probably due to
>   log files; consider pruning them.
>

These are presumably OK to delete, without any special handling for Guix?


> - Your script runs "docker commit" while guix-daemon (and other
>   programs) are still running.  To ensure the guix-daemon's database (or
>   other things) does not become corrupt, consider terminating all
>   processes before committing the new image.
>

`docker commit` pauses the container (unless you tell it not to) ...
although
I guess that could still cause problems if Guix store writes aren't
implemented
in an atomic way.

Thanks,
-SS