[Userops] Why is it hard to move from one machine to another? An analysis.

[Christopher Allan Webber]

Hello all,

For a while I've been considering, why is it so harder for me to migrate
from server to server than it is for me to migrate from desktop to
desktop?  For years, ever since I discovered rsync, migrating between
machines has not been hard.  I simply rsync my home directory over to
the new machine (or maybe even just keep the old /home/ directory's
partition where it is!) and bam, I am done.  Backing this up is easy;
it's just another rsync away.  (I use dirvish as a simple wrapper around
rsync so it can manage incremental backups.)

If I set up a new machine, it is no worry.  Even if my current machine
dies, it is *mostly* no worry.  Rsync back my home directory, and done.
I will spend a week or so discovering that certain programs I rely on
are not there, and I'll install them one by one.  In a way it's
refreshing: I can install the programs I need, and the old cruft is
gone!

This is not true for servers.  At the back of my mind I realized this,
but until the end of Stefano Zacchiroli's excellent LibrePlanet talk
when I posed a question surrounding this situation, I hadn't totally
congealed in my head: *why* is it so much harder for me to move from
server to server?  Assume I even have the old server around and I want
to move.  It isn't easy!

So here are some thoughts that come out of this:

 - For my user on my workstation, configuration and data are in the same
   place: /home/ (including lots of little dotfiles for the configs, and
   the rest is mostly data).  Sure, there's some configuration stuff in
   /etc/ and data in /var/ but it mostly doesn't really matter, and
   copying that between machines is not hard.

 - Similarly, for my user on workstation experience, it is very little
   stress if I set up a machine and am missing some common packages.  I
   can just install them again as I find them missing.

 - Neither of these are true for my server!  In addition to caring about
   /home/, and even more importantly, I have to worry about
   configuration in /etc/ and data in /var/.  These are both pains in
   the butt for me for different reasons.

 - Lots of stuff in /etc/ is configuration that interacts with the rest
   of the system in specific ways.  I could rsync it to a new machine,
   but I feel like that's just blindly copying over stuff where I really
   need to know how it *works* and how it was set up with the rest of
   the machine in the first place.

 - This is compounded by the fact that people rarely set up *one*
   machine these days; usually they have to set up several machines for
   several users.  Remembering how all that stuff worked is hard.  The
   only solution seems to be to have some sort of reproducible
   configuration system.  Hence the rise of salt, ansible, etc.  But
   these aren't really "userops" systems, they're "devops"... developer
   focused.  Not only do you need to know how they work, you need to
   know how the rest of the system works.  And it's not easy to share
   that knowledge.

 - /var/ is another matter.  Theoretically, most of my program data goes
   there (unless, of course, it went to /srv/, god help us).  But I
   can't just rsync it!  There are some processes that are very
   persnickety about the stuff there.  I have to *dump* my databases and
   etc before I can move them or back up.  Nothing sets up an automatic
   cronjob for me on these, I have to *know* to dump postgres.
   Hopefully I set up a cronjob!

 - While I as a workstation user don't stress too much if I'm missing
   some packages (just install them as I go), that is NOT true of my
   servers.  If my mail servers aren't running, if jabber isn't on, (if
   SSH isn't running!!!), there are other servers expecting to
   communicate with my machine, and if I don't set them up, I miss out.

 - Not only this, assuming I have moved between servers correctly, even
   once I have set up my machine and it has become a perfectly okay
   running special snowflake, there are certain routine tasks that
   require a lot of manual intervention, and I have only picked up the
   right steps by knowing the right friends, having run across the right
   tutorials which hopefully have shown me the right setup, etc.  SSL
   configuration, I'm looking at you; the only savior that I have is
   that I have written myself my own little orgmode notes on what to do
   the next time my certs expire.

 - My servers do become special snowflakes, and that is very stressful
   to me.  I will, in the future, need to set up one more server, and
   remembering what I did in the past will be very hard.

 - Assuming I use all the mainstream tools, not talking about "upcoming"
   ones, a better configuration management solution is probably the
   answer, right?  That's a lot to ask users though: it's not a solution
   to existing deployment, because it doesn't remove the need to know
   about all the layers underneath, it just adds a new layer to
   understand.

Those are all headaches, and they are not the only headaches.  But here
are some thoughts on things that can help:

 - If I recognize which parts of my system are "immutable" and which
   parts are "mutable", it's easier to frame how my system works.
   - /var/ is mutable, it's data.  There's no making this "reproducible"
     really: it needs to be backed up and moved around.
   - My packages and system are immutable, or mostly should be.  Even if
     not using a perfectly immutable system like guix/nix, it's helpful
     to *act* like this part of the system is pseudo-immutable, and
     simply derived from some listing of things I said I wanted
     installed.  (Nix/Guix probably do this the most nicely though.)
   - /etc/ is similarly "immutable but derived" in the best case.  I
     should be able to give the same system configuration inputs and
     always get the same system of packages and configuration files.

 - I like Guix/Nix, but my usage of Debian and Fedora and friends is not
   going away anytime soon.  Nonetheless, configuration management
   systems like puppet/ansible/salt help give the *illusion* of an
   immutable system derived from a set of inputs, even though they are
   working within a mutable one.

 - Language packaging for deployment needs to die.  Yes, I say this as a
   project that advocates that very route.  We're doing it wrong, and
   I want to change it.
   (Language packaging for development though: that's great!)

 - Asking people to use systems like ansible/salt/puppet is asking users
   too much.  You're just asking them to learn one more layer on top of
   knowing how the whole system works.  Sharing common code is mostly
   copy and paste.  There are some layers built on top of here to
   mitigate this but afaict they aren't really good, not good enough.
   (I am working on something to solve this...)

 - Pre-built containers are not the solution.  Sorry container people!
   Containers can be really useful but only if they are built in some
   reproducible way.  But very few people using Docker and etc seem to
   be doing this.  But here's another thing: Docker and friends contain
   their own deployment domain specific languages, which is dumb.  If a
   reproducible configuration system is good enough, it should be good
   enough for a VM or a container or a vanilla server or a desktop.  So
   maybe we can use containers as lightweight and even sandboxed VMs,
   but we shouldn't be installing prebuilt containers on our servers
   alone as a system.

   Otherwise else you're running 80 heavy and expensive Docker images
   that slowly go out of date... now you're not maintaining 1
   distribution install, you're maintaining 81 of them.  Yikes!  Good
   luck with the next Shellshock!

 - Before Asheesh jumps in here: yes I will say that Sandstorm is taking
   maybe the best route as in terms of a system that uses containers
   heavily (and unlike Docker, they seem actually sandboxed) in that it
   seems to have a separation between mutable parts and immutable parts:
   the container is more or less an immutable machine from what I can
   tell that has /var/ mounted *into* it, which is a pretty good route.

   In this sense I think Sandstorm has a good picture of things.  There
   are other things that I am still very unsure about, and Asheesh knows
   because I have expressed them to him (I sure hope that iframe thing
   goes away, and that daemons like Celery can run, and etc!) but at
   least in this sense, Sandstorm's container story is more sane.

So there are some reflections in case you are planning on debugging why
these things are hard.

 - Chris

PS: If you haven't gotten the sense, the direction I'm thinking of is
more along the lines of Guix becoming our Glorious Future (TM) assuming
something like GuixOps can happen (go Dave Thompson, go Guix crew!) and
a web UI can be built on top of it with some sort of common recipe
system.

But I don't think our imperative systems like Debian are going away
anytime soon; I certainly don't intend to move all my stuff over to Guix
at this time.  For that reason, I think there needs to be another
program to fit the middle ground: something like salt/ansible/puppet,
but with less insane one-off domain specific languages, with a sharable
recipe system, and scalable both from developer-oriented scripts but
also having a user-friendly web interface.  I've begun working on this
tool, and it's called Opstimal.  Expect to hear more about it soon.