Re: It's time to build "guix deploy"

[Christopher Lemmer Webber]

Thompson, David writes:

> Hi Chris,
>
> Here we go again, eh? :)
>
>>   https://lists.gnu.org/archive/html/guix-devel/2015-05/msg00007.html
>>   https://lists.gnu.org/archive/html/guix-devel/2015-06/msg00006.html
>>
>> There is a heavily, heavily bitrotted branch named "wip-deploy" where
>> David originally started exploring these ideas.  Conveniently, it's all
>> in one commit:
>>
>>   
>> https://git.savannah.gnu.org/cgit/guix.git/commit/?h=wip-deploy&id=fcd6fc84e493d05be1f7590ee77509c81ac315c2
>
> Useful for context, but the code can probably be tossed at this point.

OK!

>> That seems like a good starting point.  But I know David feels that with
>> real-world experience in devops type work that actually things would be
>> a bit different than what's described in his original email.  I'm not
>> sure myself what would be different.  It would be helpful to hear Dave
>> weigh in on that point.
>
> Sure, since 2015 I've become the lead devops person at my company, so
> I like to think I'm a bit wiser now.

:)

>> Maybe Dave and I can meet up IRL now that we're close enough to each
>> other to chat about it.  But I know it's less fun than it used to be for
>> Dave to consider this because now that's Dave's actual job... but all
>> the more reason we need Dave's wisdom! :)
>
> We could meet up IRL about this and I can try to make an earnest
> effort to deal with this. I think what has stopped me in the past is
> the sheer size of this project, and maybe dramatically scaling down
> the scope will allow us to get *something* out the door.

Yes I think we shouldn't allow ourselves to get too overwhelmed!  We
should give the warnings that this is an alpha system in alpha (soon
beta?) software and we don't guarantee that the api will be stable :)

Furthermore, I think if we put an emphasis on composable layers, and
start with the simplest lower layers, and then add some other layers
above it (and actually the different use cases you lay out may actually
swap out which layers are above this), then I think we could be good?

> Here are some general use-cases I know about for deployments, roughly
> ordered from small scale to large scale, and least complex to most
> complex:
>
> * Managing a physical machine or two that have been given memorable
> names that you update in-place (home scale)
> * Managing a virtual machine or two that have been given memorable
> names that you update in-place (blog scale)

In the past we have discussed storing "deployment state", and where and
how we would store it.  I will make the argument that for these two
cases, we actually don't need state!  We can just do the simplest thing
possible: build the system derivation for each of these machines, log in
and see if the system already has that derivation running, and if not
copy over the closure and install it as the system state.  This could
be entirely declarative.

I think initially we can ignore entirely the issue of provisioning an
initial guix image.  We can also ignore things like getting the right
public keys and configuring the initial authority of how you can be
authorized to copy over that data.

> * Managing a large number of virtual machines whose names don't matter
> that you update in-place (proto-cloud scale)
> * Managing a large number of virtual machines whose names don't matter
> that are replaced when there is an update (cloud scale)
> * Managing 1 or more clusters of physical machines (datacenter scale)
> * Managing 1 or more clusters of physical machines and virtual
> machines ("corporation with a datacenter that is moving some stuff to
> the cloud" scale)

For all these, I think there can be one or more abstraction layers that
build on top of the same declarative primitives described above.  And
this may be where state comes in; since you don't really know what the
identifiers are ahead of time, but you do need to keep track of them,
yes we do want to serialize the state somewhere.

> There are, of course, more scenarios to consider (haven't even touched
> upon things like a Kubernetes cluster), but this is enough to
> illustrate the point that is a great diversity in setups.  How many
> machines are there? Are the bare metal, virtual machines, or a mix of
> both? In the case of virtual machines, are updates applied in an
> immutable fashion or not?  If immutable, which technique (blue-green,
> rolling release, etc.)?  It makes my head spin to think about all the
> use-cases.

... I don't know anything about the things you said above so I guess
that would be nice to go over in person :)

I think maybe my ignorance is helpful for optimism in this case!

> So... let's start small. Can we write a tool that handles in-place
> updates to machines (physical or virtual) whose name and IP address we
> know well (our special pet servers) without precluding the possibility
> of scaling up to more sophisticated architectures?  This would address
> the "home" and "blog" scale items above, which is probably what most
> of the people actually using Guix today would want.  I got stuck
> trying to do in-place updates to remote machines years ago, but that
> was before Ludo made it easy to connect to remote systems.

Yes!

Also I think guile-ssh might make things a bit easier than they were in
the past, too.

Let's do this!