Re: [Userops] Why is it hard to move from one machine to another? An analysis.

[韋嘉誠]

On Apr 10, 2015 5:03 PM, "Dave Crossland" <address@hidden> wrote:
> On 10 April 2015 at 09:02, Claes Wallin (韋嘉誠) <address@hidden> wrote:

>> Going briefly off-topic, I'm curious when "post-VM" VMs are going to
>> start happening. There's a lot of gooey hardware-management stuff
>> happening in VMs that don't have actual hardware, where it should all
>> just be what amounts to APIs to the hypervisor.
>
> I think that's a great way to think about containers: They segment bare metal without the needless software-defined hardware interfaces of VMs.

Well, they're on different levels. A container can provide a higher-level interface -- you're in Linux when you're in Docker, there's a kernel API and there's a filesystem and networking. But if that particular abstraction is not what you want or need, a hypervisor provides you with a lower-level abstraction with maybe just memory and CPU and a network card abstraction.

>> Actually it's tangentially on-topic, because Mirage OS in particular
>> is also about deployment -- your service is just one binary that you
>> run right in your hypervisor.
>>
>> [1] http://www.openmirage.org/
>
>
> Hmm. This is Xen based. 
>
> Xen is a hypervisor, not a VM or a container.

A VM is a broad term that includes virtualized hardware, emulated hardware, blends of the two and a byte-compiled environment like Smalltalk, the Java VM or the CLR.

Xen is one hypervisor that can run VMs, VMWare, VirtualBox, Parallels and Linux's native KVM are others. Xen is the only one of these that handles the bare hardware itself, the others run inside full OSes.

All of these provide the VM with a blend of virtualized hardware, which runs native and only traps certain operations to isolate the VM from the hypervisor or host OS, emulations of existing hardware, like Cirrus graphics cards or Intel network cards, and emulations of virtual hardware that don't correspond to any existing hardware, like the virtio SATA and network drivers.

> As I understand things: 
>
> - VMs virtualise hardware completely in software. You can run software compiled for another CPU architecture in a VM, eg Super Marios in a NES emulator runs on your laptop.

That's a VM running in a full system emulation, like qemu when it runs e.g. an ARM VM on an x86 system.

That's by definition not virtualization, which is hardware-aided isolation from the host. It's emulation, but the emulated machine is still called a virtual machine, because it's not a physical machine.

> - Hypervisors like Xen (and KVM, which superceded it; and vmware's vSphere and msft's Hyper-V) virtualise hardware using 'pass through' CPU hardware, to get near-native speeds. Your guest OSs need to be compiled for the same CPU architecture, but you can run heterogenous OSs, eg Windows on KVM or Debian on Hyper-V.

Yep, that's virtualization. Like you say, it allows you to run multiple instances of kernel-mode code in one physical machine.

Xen has not been superceded by kvm, they both live on and are being developed. Maybe Xen has been kicked out of mainline Linux though, I haven't checked.

If you have a machine that lacks the vmx extension, say a Core i3 laptop, Xen can't run x86-platform OS:es and KVM can't do a thing, but Xen can still run Xen-platform OS:es, e.g. a Linux configured and compiled for Xen.

> - Containers virtualise as much as everything above the kernel, but potentially just one program, using 'overlay filesystems' software, to get native speeds.

Containers don't virtualize, hardwate doesn't play into it, they provide an isolated userspace environment with APIs to the kernel. They're modern-era jails with improved isolation and improved abilities to constrain and monitor resource use.

> It becomes increasingly easy to punch out of the virual environment to get root on the bare metal as the amount of virtualisation decreases.

The Linux API is more complex and therefore likely more full of holes than the Xen abstraction. So I'd say Xen VMs are likely to be better isolated from each other than LXC process groups are. Either way, once you break out, you control the machine. And you only need one hole to break out.

If you mean "virtualization decreases" as in, I could run a VM and then run Docker inside that, then sure, that's one more layer you need to find a hole in before you reach my physical machine.

> Is this correct?

More or less, I'm just picky with the terminology. ;-)

> However, MirageOS does look very nice; I think of it as yet another example of what I pointed to earlier with https://us.pycon.org/2015/schedule/presentation/378/ and what Shawn Landden said,

One cool thing with MirageOS is they saved a lot of effort on handling x86 platform hardware, because they were targeting Xen, not metal. When you are targeting a physical legacy platform you have to handle a lot of quirks and the hypervisor has to handle them too. Xen offers a cleaner interface, which at least in theory has performance advantages and in practice offers an easier target if you're building from scratch anyway.

>>  go binaries are statically linked, and if the application is designed right could go into a container with no other files.

Yeah, that's what people should do more.

> I don't really see the difference between a MirageOS host and a RancherOS or CoreOS host, other than optimisation (maybe prematurely ;)

CoreOS and RancherOS run on Linux.

MirageOS has nothing to do with Linux, it's OCaml all the way from the hypervisor down to your code. There is no separate kernel really, that's why they call it a unikernel. You link the parts of the OS that you need right into the same binary as your own code and then you run that binary right on the Xen "metal". Every binary you produce is its own OS with only the parts you need, and it's all OCaml, so you get some cool benefits from the type system in terms of correctness and optimizations.

It's pretty far off to consider MirageOS anywhere near something that could be described as userops though, so maybe we should round off this detour. :-)