Re: [orchestration] AWS public cloud images

[Mark Meyer]

Hi David,
I think most of the AWS build processes feel imperative and hacky. I always 
equate this to a big pot of soup which you stir and pass on to the next guy, 
who builds another image/stirs it again. In general I think you should not 
start with an AMI, but with an empty volume and then deploy to that. Your AMI 
definition process should be composable, so if you have different teams working 
on it, i.e. systems engineering builds the base AMI and an application group 
customizes it, you should only run one AMI build and not pass down the 
artifact. To put it more extreme: it  looks like some junkies passing around a 
dirty needle.

That's one of the reasons the code to build the AMIs is currently the main 
deliverable. My proposition is not to refrain from providing a Golden Master 
image, it will be worthwhile for some purposes, my suggestion is to make the 
AMI build process fully automated and easily extensible.

For the moment I don't mind having Packer in the mix. I don't however hold it 
in high regards, I think it's essentially glue code. But when you want to 
replace it, please consider that there are use cases like different partitions 
(GovCloud and AWS China) that have very different implementations of the AWS 
API. Speaking from my experience these can lead to huge headaches, and in 
general I think it's not worth to work around the same set of bugs another time.

Cheers, Mark

On Tue, Mar 27, 2018, at 20:08, Thompson, David wrote:
> Hi Mark,
> 
> On Mon, Mar 26, 2018 at 4:18 PM, Mark Meyer <address@hidden> wrote:
> > Hi,
> > I've the beginning of Guix cloud images available over at Github at
> >
> >   https://github.com/ofosos/guix-packer/
> >
> > There's a small writeup of what has been done and what's still missing over 
> > here:
> >
> >   https://ofosos.org/2018/03/26/guix-images-01/
> >
> > All in all, I split the heavy lifting between Packer (AWS API calls) and 
> > `guix system` and that worked remarkably well. There's some integration 
> > with EC2 (You can inject a pubkey into the image via the console), but 
> > there's also a lot of stuff missing. In the end, I would like to provide 
> > public cloud images in some weeks time. Of course you'll have all the tools 
> > to rebuild your own images (surprisingly simple).
> >
> > I think there's still a lot of polish we can apply, but at some point we'll 
> > need some help from AWS. I do have AWS support access at work, but am not 
> > really comfortable to use company resources for this yet, but I'll probably 
> > ask if there's some  avenue to get some 'official' help as a free software 
> > project, when I run across our technical account manager.
> 
> First of all: Thanks! This is a great start! I've wanted to run GuixSD
> EC2 instances for some time but haven't gotten around to it. The more
> I think about it, though, the more I wonder how useful an official
> GuixSD image is vs. providing software to create AMIs for any given
> system configuration. The use-case I'm particularly interested in is
> using GuixSD instances in auto scaling groups.  I'm wondering if
> there's any other way to create GuixSD AMIs than starting from some
> official image, updating Guix, running 'guix system reconfigure', and
> using the result as the basis for the AMI. While this ought to work,
> it feels very imperative and hacky.  In the future it would also be
> great to eliminate the need for Packer entirely and replace it with
> Guile.
> 
> Anyway, just some food for thought.  Awesome work!
> 
> - Dave


-- 
  Mark Meyer
  address@hidden