Re: [Qemu-discuss] troubleshooting live migration

[Jakob Bohm]

Just one small idea:

Could it be that the network access is lost because some (physical?)
networkswitch is not accepting that packets to/from the guest VMs
virtual MACaddress shall now be sent to the physical switch port
connected to host2,and not the one connected to host1?

This obviously wold not directly explain the CPU thrashing, but that
might bea secondary effect caused by the loss of network
connectivity.

Of cause the converse might also be true: the network being fine, but
the thrashing CPU preventing the VM from answering pings etc.

On 1/14/2014 8:23 AM, Marcus Sorensen wrote:
> I suppose I can capture a core while it's spinning, if someone is
> willing to look at it.
>
> On Tue, Jan 14, 2014 at 12:20 AM, Marcus Sorensen <address@hidden> wrote:
> > Does anyone have tips on troubleshooting live migration? I've got
> > several E5-2650 servers running in test environment, kernel 3.10.26
> > and qemu 1.7.0. If I start a VM guest (say ubuntu, debian, or centos),
> > I can migrate it around  from host to host to host just fine, but if I
> > wait awhile (say 1 hour), I try to migrate and it succeeds but the
> > guest is hosed. No longer pings, cpu is thrashing. I've tried to
> > strace it and don't see anything that other working hosts aren't
> > doing, and I've tried gdb but I'm not entirely sure what I'm doing. I
> > tried downgrading to qemu 1.6.1. I've found dozens of reports of such
> > behavior, but they're all due to other things (migrating between
> > different host CPUs, someone thinking it's virtio or memballoon only
> > to later find a fix like changing machine type, etc). I'm at a loss.
> > This seems to work just fine with stock CentOS builds.
> >
> > Here's an example xml:
> >
> > <domain type='kvm'>
> >    <name>VM12</name>
> >    <uuid>dd25acfc-e24d-4de6-814c-72ac465bc208</uuid>
> >    <description></description>
> >    <memory unit='KiB'>4194304</memory>
> >    <currentMemory unit='KiB'>4194304</currentMemory>
> >    <vcpu placement='static'>2</vcpu>
> >    <cputune>
> >      <shares>2000</shares>
> >    </cputune>
> >    <resource>
> >      <partition>/machine</partition>
> >    </resource>
> >    <os>
> >      <type arch='x86_64' machine='pc-i440fx-1.7'>hvm</type>
> >      <boot dev='cdrom'/>
> >      <boot dev='hd'/>
> >    </os>
> >    <features>
> >      <acpi/>
> >      <apic/>
> >      <pae/>
> >    </features>
> >    <cpu>
> >    </cpu>
> >    <clock offset='utc'>
> >      <timer name='kvmclock' tickpolicy='catchup'/>
> >    </clock>
> >    <on_poweroff>destroy</on_poweroff>
> >    <on_reboot>restart</on_reboot>
> >    <on_crash>destroy</on_crash>
> >    <devices>
> >      <emulator>/usr/bin/qemu-kvm</emulator>
> >      <disk type='block' device='disk'>
> >        <driver name='qemu' type='raw' cache='none'/>
> >        <source dev='/dev/sdc'/>
> >        <target dev='vda' bus='virtio'/>
> >        <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> > function='0x0'/>
> >      </disk>
> >      <disk type='file' device='cdrom'>
> >        <driver name='qemu' type='raw' cache='none'/>
> >        <target dev='hdc' bus='ide'/>
> >        <readonly/>
> >        <address type='drive' controller='0' bus='1' target='0' unit='0'/>
> >      </disk>
> >      <controller type='ide' index='0'>
> >        <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > function='0x1'/>
> >      </controller>
> >      <controller type='virtio-serial' index='0'>
> >        <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> > function='0x0'/>
> >      </controller>
> >      <controller type='usb' index='0'>
> >        <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> > function='0x2'/>
> >      </controller>
> >      <controller type='pci' index='0' model='pci-root'/>
> >      <interface type='bridge'>
> >        <mac address='02:00:09:66:00:18'/>
> >        <source bridge='br1000192'/>
> >        <model type='virtio'/>
> >        <bandwidth>
> >          <inbound average='128000' peak='128000'/>
> >          <outbound average='128000' peak='128000'/>
> >        </bandwidth>
> >        <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> > function='0x0'/>
> >      </interface>
> >      <serial type='pty'>
> >        <target port='0'/>
> >      </serial>
> >      <console type='pty'>
> >        <target type='serial' port='0'/>
> >      </console>
> >      <channel type='unix'>
> >        <source mode='bind' path='/var/lib/libvirt/qemu/VM12.agent'/>
> >        <target type='virtio' name='VM12.vport'/>
> >        <address type='virtio-serial' controller='0' bus='0' port='1'/>
> >      </channel>
> >      <input type='tablet' bus='usb'/>
> >      <input type='mouse' bus='ps2'/>
> >      <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
> >        <listen type='address' address='0.0.0.0'/>
> >      </graphics>
> >      <video>
> >        <model type='cirrus' vram='9216' heads='1'/>
> >        <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> > function='0x0'/>
> >      </video>
> >      <memballoon model='virtio'>
> >        <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> > function='0x0'/>
> >      </memballoon>
> >    </devices>
> >    <seclabel type='none'/>
> > </domain>


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S.  http://www.wisemo.com
Transformervej 29, 2730 Herlev, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded