qemu-discuss
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-discuss] Guests freeze after migration

From: Magnus
Subject: [Qemu-discuss] Guests freeze after migration
Date: Tue, 17 Sep 2013 13:33:18 +0200
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130803 Thunderbird/17.0.8 
Hello,
I am trying to run libvirt on two nodes (debian wheezy) with the intention of being able to migrate guests between them. I am using glusterfs as the backend storage with a volume replicated on both servers.
The gluster volume is mounted on each node and I have a directory pool in libvirt for that.
Currently glusterfs-3.4, libvirt-bin 0.9.12, qemu-1.1.2, linux-image-3.10 from wheezy-backports.
The problem I have is that migrations from node B to node A work but migration from node A to node B always fails. virt-manager says the guest is running on the target (B) but shows a constant high cpu usage and the console is completly frozen. This happens with both debian wheezy guests and a test FreeBSD-9.1 guest since I wanted make sure it wasn't a problem with wheezy.
I have also tried running ubuntu 13.04 with the exact same symptoms, glusterfs versions from 3.2.7 to 3.4 and qemu up to 1.5.
I cleared libvirtd.log and the qemu log for this specific guest and did a migration see logs below. 

I really could use some help in troubleshooting this further.

Best Regards
Magnus

Node A (8 cores)
------------------------
processor       : 7
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 2
model name      : AMD FX(tm)-8320 Eight-Core Processor
stepping        : 0
microcode       : 0x600081f
cpu MHz         : 1400.000
cache size      : 2048 KB
physical id     : 0
siblings        : 8
core id         : 7
cpu cores       : 4
apicid          : 23
initial apicid  : 7
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm topoext perfctr_core perfctr_nb arat cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold bmi1 
bogomips        : 7023.38
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro



Node B (6 cores)
------------------------
processor       : 5
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 10
model name      : AMD Phenom(tm) II X6 1055T Processor
stepping        : 0
microcode       : 0x10000dc
cpu MHz         : 2800.000
cache size      : 512 KB
physical id     : 0
siblings        : 6
core id         : 5
cpu cores       : 6
apicid          : 5
initial apicid  : 5
fpu             : yes
fpu_exception   : yes
cpuid level     : 6
wp              : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt cpb hw_pstate npt lbrv svm_lock nrip_save pausefilter 
bogomips        : 5624.50
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate cpb


Node A libvirtd.log
-------------------------------------
2013-09-17 08:58:06.885+0000: 12476: error : virNetServerProgramDispatchCall:384 : unknown procedure: 293 2013-09-17 08:58:06.941+0000: 12474: error : virNetServerProgramDispatchCall:384 : unknown procedure: 293 2013-09-17 08:58:07.032+0000: 12474: error : virNetServerProgramDispatchCall:384 : unknown procedure: 293 2013-09-17 08:58:27.043+0000: 12475: error : qemuDomainMigrateSetMaxDowntime:9488 : invalid argument: qemuDomainMigrateSetMaxDowntime: unsupported fl 
ags (0xbc614e)
2013-09-17 08:58:27.144+0000: 12477: error : qemuDomainMigrateSetMaxDowntime:9488 : invalid argument: qemuDomainMigrateSetMaxDowntime: unsupported fl 
ags (0xbc614e)
2013-09-17 08:59:02.021+0000: 12473: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor 
without asking for a nested job is dangerous
2013-09-17 08:59:02.173+0000: 12472: error : qemuMonitorIORead:522 : Unable to read from monitor: Connection reset by peer 2013-09-17 08:59:07.157+0000: 12477: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor 
without asking for a nested job is dangerous
2013-09-17 08:59:07.301+0000: 12477: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor 
without asking for a nested job is dangerous
2013-09-17 08:59:07.303+0000: 12477: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor 
without asking for a nested job is dangerous
2013-09-17 08:59:07.371+0000: 12477: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor 
without asking for a nested job is dangerous
2013-09-17 08:59:07.371+0000: 12477: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor 
without asking for a nested job is dangerous


Node A qemu log
-------------------------------------
2013-09-17 08:49:59.291+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=20,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-p 
ci,id=balloon0,bus=pci.0,addr=0x4
2013-09-17 08:49:59.313+0000: 12377: info : libvirt version: 0.9.12
2013-09-17 08:49:59.313+0000: 12377: debug : virCommandHook:2041 : Run hook 0x48e9e0 0x7f97d45bd3f0 2013-09-17 08:49:59.313+0000: 12377: debug : qemuProcessHook:2560 : Obtaining domain lock 2013-09-17 08:49:59.313+0000: 12377: debug : virDomainLockManagerNew:123 : plugin=0x7f97c800b210 dom=0x7f97c815d960 withResources=1 2013-09-17 08:49:59.313+0000: 12377: debug : virLockManagerNew:291 : plugin=0x7f97c800b210 type=0 nparams=4 params=0x7f97d45bc950 flags=0 2013-09-17 08:49:59.313+0000: 12377: debug : virLockManagerLogParams:98 : key=uuid type=uuid value=15f7c568-e477-43cd-20da-4b3830643b43 2013-09-17 08:49:59.313+0000: 12377: debug : virLockManagerLogParams:94 : key=name type=string value=FreeBSD 2013-09-17 08:49:59.313+0000: 12377: debug : virLockManagerLogParams:82 : key=id type=uint value=1 2013-09-17 08:49:59.313+0000: 12377: debug : virLockManagerLogParams:82 : key=pid type=uint value=12377 2013-09-17 08:49:59.314+0000: 12377: debug : virDomainLockManagerNew:135 : Adding leases 2013-09-17 08:49:59.314+0000: 12377: debug : virDomainLockManagerNew:140 : Adding disks 2013-09-17 08:49:59.314+0000: 12377: debug : virDomainLockManagerAddDisk:86 : Add disk /vmimages/gv0/FreeBSD.img 2013-09-17 08:49:59.314+0000: 12377: debug : virLockManagerAddResource:320 : lock=0x1b9b170 type=0 name=/vmimages/gv0/FreeBSD.img nparams=0 params=(n 
il) flags=0
2013-09-17 08:49:59.314+0000: 12377: debug : virLockManagerAcquire:337 : lock=0x1b9b170 state='(null)' flags=3 fd=0x7f97d45bca34 2013-09-17 08:49:59.314+0000: 12377: debug : virLockManagerFree:374 : lock=0x1b9b170 2013-09-17 08:49:59.314+0000: 12377: debug : qemuProcessHook:2585 : Moving procss to cgroup 2013-09-17 08:49:59.314+0000: 12377: debug : qemuProcessHook:2611 : Setup CPU affinity 2013-09-17 08:49:59.314+0000: 12377: debug : qemuProcessInitCpuAffinity:1808 : Setting CPU affinity 2013-09-17 08:49:59.315+0000: 12377: debug : qemuProcessInitCpuAffinity:1844 : Set CPU affinity with specified cpuset 2013-09-17 08:49:59.316+0000: 12377: debug : qemuProcessHook:2618 : Setting up security labelling 2013-09-17 08:49:59.316+0000: 12377: debug : virSecurityDACSetProcessLabel:642 : Dropping privileges of DEF to 105:111 2013-09-17 08:49:59.316+0000: 12377: debug : qemuProcessHook:2625 : Hook complete ret=0 2013-09-17 08:49:59.316+0000: 12377: debug : virCommandHook:2043 : Done hook 0 2013-09-17 08:49:59.316+0000: 12377: debug : virCommandHook:2056 : Notifying parent for handshake start on 23 2013-09-17 08:49:59.316+0000: 12377: debug : virCommandHook:2077 : Waiting on parent for handshake complete on 24 2013-09-17 08:49:59.322+0000: 12377: debug : virCommandHook:2093 : Hook is done 0 
char device redirected to /dev/pts/1
2013-09-17 08:50:56.556+0000: shutting down
qemu: terminating on signal 15 from pid 12304
2013-09-17 08:51:13.112+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=20,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -incoming tcp:0.0.0.0:49 
152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
2013-09-17 08:51:13.116+0000: 12404: info : libvirt version: 0.9.12
2013-09-17 08:51:13.116+0000: 12404: debug : virCommandHook:2041 : Run hook 0x48e9e0 0x7f97d45bd320 2013-09-17 08:51:13.116+0000: 12404: debug : qemuProcessHook:2560 : Obtaining domain lock 2013-09-17 08:51:13.116+0000: 12404: debug : virDomainLockManagerNew:123 : plugin=0x7f97c800b210 dom=0x7f97c815d960 withResources=1 2013-09-17 08:51:13.116+0000: 12404: debug : virLockManagerNew:291 : plugin=0x7f97c800b210 type=0 nparams=4 params=0x7f97d45bc880 flags=0 2013-09-17 08:51:13.116+0000: 12404: debug : virLockManagerLogParams:98 : key=uuid type=uuid value=15f7c568-e477-43cd-20da-4b3830643b43 2013-09-17 08:51:13.116+0000: 12404: debug : virLockManagerLogParams:94 : key=name type=string value=FreeBSD 2013-09-17 08:51:13.116+0000: 12404: debug : virLockManagerLogParams:82 : key=id type=uint value=2 2013-09-17 08:51:13.116+0000: 12404: debug : virLockManagerLogParams:82 : key=pid type=uint value=12404 2013-09-17 08:51:13.116+0000: 12404: debug : virDomainLockManagerNew:135 : Adding leases 2013-09-17 08:51:13.116+0000: 12404: debug : virDomainLockManagerNew:140 : Adding disks 2013-09-17 08:51:13.116+0000: 12404: debug : virDomainLockManagerAddDisk:86 : Add disk /vmimages/gv0/FreeBSD.img 2013-09-17 08:51:13.116+0000: 12404: debug : virLockManagerAddResource:320 : lock=0x7f97c0059ea0 type=0 name=/vmimages/gv0/FreeBSD.img nparams=0 para 
ms=(nil) flags=0
2013-09-17 08:51:13.116+0000: 12404: debug : virLockManagerAcquire:337 : lock=0x7f97c0059ea0 state='(null)' flags=3 fd=0x7f97d45bc964 2013-09-17 08:51:13.116+0000: 12404: debug : virLockManagerFree:374 : lock=0x7f97c0059ea0 2013-09-17 08:51:13.116+0000: 12404: debug : qemuProcessHook:2585 : Moving procss to cgroup 2013-09-17 08:51:13.116+0000: 12404: debug : qemuProcessHook:2611 : Setup CPU affinity 2013-09-17 08:51:13.116+0000: 12404: debug : qemuProcessInitCpuAffinity:1808 : Setting CPU affinity 2013-09-17 08:51:13.118+0000: 12404: debug : qemuProcessInitCpuAffinity:1844 : Set CPU affinity with specified cpuset 2013-09-17 08:51:13.118+0000: 12404: debug : qemuProcessHook:2618 : Setting up security labelling 2013-09-17 08:51:13.118+0000: 12404: debug : virSecurityDACSetProcessLabel:642 : Dropping privileges of DEF to 105:111 2013-09-17 08:51:13.119+0000: 12404: debug : qemuProcessHook:2625 : Hook complete ret=0 2013-09-17 08:51:13.119+0000: 12404: debug : virCommandHook:2043 : Done hook 0 2013-09-17 08:51:13.119+0000: 12404: debug : virCommandHook:2056 : Notifying parent for handshake start on 22 2013-09-17 08:51:13.119+0000: 12404: debug : virCommandHook:2077 : Waiting on parent for handshake complete on 23 2013-09-17 08:51:13.125+0000: 12404: debug : virCommandHook:2093 : Hook is done 0 
char device redirected to /dev/pts/1
inet_listen_opts: bind(ipv4,0.0.0.0,49152): Address already in use
inet_listen_opts: FAILED
Migrate: Failed to bind socket
Migration failed. Exit code tcp:0.0.0.0:49152(-1), exiting.
2013-09-17 08:51:13.571+0000: shutting down
2013-09-17 08:51:16.886+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=20,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -incoming tcp:0.0.0.0:49 
153 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
2013-09-17 08:51:16.891+0000: 12424: info : libvirt version: 0.9.12
2013-09-17 08:51:16.891+0000: 12424: debug : virCommandHook:2041 : Run hook 0x48e9e0 0x7f97d45bd320 2013-09-17 08:51:16.891+0000: 12424: debug : qemuProcessHook:2560 : Obtaining domain lock 2013-09-17 08:51:16.891+0000: 12424: debug : virDomainLockManagerNew:123 : plugin=0x7f97c800b210 dom=0x7f97c815d960 withResources=1 2013-09-17 08:51:16.891+0000: 12424: debug : virLockManagerNew:291 : plugin=0x7f97c800b210 type=0 nparams=4 params=0x7f97d45bc880 flags=0 2013-09-17 08:51:16.891+0000: 12424: debug : virLockManagerLogParams:98 : key=uuid type=uuid value=15f7c568-e477-43cd-20da-4b3830643b43 2013-09-17 08:51:16.891+0000: 12424: debug : virLockManagerLogParams:94 : key=name type=string value=FreeBSD 2013-09-17 08:51:16.891+0000: 12424: debug : virLockManagerLogParams:82 : key=id type=uint value=3 2013-09-17 08:51:16.891+0000: 12424: debug : virLockManagerLogParams:82 : key=pid type=uint value=12424 2013-09-17 08:51:16.891+0000: 12424: debug : virDomainLockManagerNew:135 : Adding leases 2013-09-17 08:51:16.891+0000: 12424: debug : virDomainLockManagerNew:140 : Adding disks 2013-09-17 08:51:16.891+0000: 12424: debug : virDomainLockManagerAddDisk:86 : Add disk /vmimages/gv0/FreeBSD.img 2013-09-17 08:51:16.891+0000: 12424: debug : virLockManagerAddResource:320 : lock=0x7f97c005b1f0 type=0 name=/vmimages/gv0/FreeBSD.img nparams=0 para 
ms=(nil) flags=0
2013-09-17 08:51:16.891+0000: 12424: debug : virLockManagerAcquire:337 : lock=0x7f97c005b1f0 state='(null)' flags=3 fd=0x7f97d45bc964 2013-09-17 08:51:16.891+0000: 12424: debug : virLockManagerFree:374 : lock=0x7f97c005b1f0 2013-09-17 08:51:16.891+0000: 12424: debug : qemuProcessHook:2585 : Moving procss to cgroup 2013-09-17 08:51:16.891+0000: 12424: debug : qemuProcessHook:2611 : Setup CPU affinity 2013-09-17 08:51:16.891+0000: 12424: debug : qemuProcessInitCpuAffinity:1808 : Setting CPU affinity 2013-09-17 08:51:16.893+0000: 12424: debug : qemuProcessInitCpuAffinity:1844 : Set CPU affinity with specified cpuset 2013-09-17 08:51:16.893+0000: 12424: debug : qemuProcessHook:2618 : Setting up security labelling 2013-09-17 08:51:16.893+0000: 12424: debug : virSecurityDACSetProcessLabel:642 : Dropping privileges of DEF to 105:111 2013-09-17 08:51:16.894+0000: 12424: debug : qemuProcessHook:2625 : Hook complete ret=0 2013-09-17 08:51:16.894+0000: 12424: debug : virCommandHook:2043 : Done hook 0 2013-09-17 08:51:16.894+0000: 12424: debug : virCommandHook:2056 : Notifying parent for handshake start on 22 2013-09-17 08:51:16.894+0000: 12424: debug : virCommandHook:2077 : Waiting on parent for handshake complete on 23 2013-09-17 08:51:16.900+0000: 12424: debug : virCommandHook:2093 : Hook is done 0 
char device redirected to /dev/pts/1
2013-09-17 08:51:36.593+0000: shutting down
qemu: terminating on signal 15 from pid 12304
2013-09-17 08:57:52.843+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=20,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device virtio-balloon-p 
ci,id=balloon0,bus=pci.0,addr=0x4
char device redirected to /dev/pts/1
2013-09-17 08:58:48.458+0000: shutting down
qemu: terminating on signal 15 from pid 12472
2013-09-17 08:59:01.807+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=20,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -incoming tcp:0.0.0.0:49 
152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
char device redirected to /dev/pts/1
inet_listen_opts: bind(ipv4,0.0.0.0,49152): Address already in use
inet_listen_opts: FAILED
Migrate: Failed to bind socket
Migration failed. Exit code tcp:0.0.0.0:49152(-1), exiting.
2013-09-17 08:59:02.173+0000: shutting down
2013-09-17 08:59:06.943+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=20,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -incoming tcp:0.0.0.0:49 
153 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
char device redirected to /dev/pts/1


Node B libvirtd.log
-------------------------------------
2013-09-17 08:58:34.981+0000: 18139: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2013-09-17 08:58:35.414+0000: 18134: error : qemuMonitorIORead:522 : Unable to read from monitor: Connection reset by peer 2013-09-17 08:58:46.597+0000: 18135: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2013-09-17 08:58:47.088+0000: 18135: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2013-09-17 08:58:47.091+0000: 18135: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2013-09-17 08:58:47.170+0000: 18135: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2013-09-17 08:58:47.170+0000: 18135: warning : qemuDomainObjEnterMonitorInternal:994 : This thread seems to be the async job owner; entering monitor without asking for a nested job is dangerous 2013-09-17 08:58:49.143+0000: 18139: error : virNetServerProgramDispatchCall:384 : unknown procedure: 293 2013-09-17 08:58:49.203+0000: 18136: error : virNetServerProgramDispatchCall:384 : unknown procedure: 293 2013-09-17 08:58:58.933+0000: 18136: error : qemuDomainMigrateSetMaxDowntime:9488 : invalid argument: qemuDomainMigrateSetMaxDowntime: unsupported flags (0xbc614e) 2013-09-17 08:58:58.997+0000: 18139: error : qemuDomainMigrateSetMaxDowntime:9488 : invalid argument: qemuDomainMigrateSetMaxDowntime: unsupported flags (0xbc614e) 


Node B qemu.log
-------------------------------------
2013-09-17 08:50:50.192+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=31,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:11 -vga cirrus -incoming tcp:0.0.0.0:4 
9152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
2013-09-17 08:50:50.198+0000: 17797: info : libvirt version: 0.9.12
2013-09-17 08:50:50.198+0000: 17797: debug : virCommandHook:2041 : Run hook 0x48e9e0 0x7fc3103a4320 2013-09-17 08:50:50.198+0000: 17797: debug : qemuProcessHook:2560 : Obtaining domain lock 2013-09-17 08:50:50.198+0000: 17797: debug : virDomainLockManagerNew:123 : plugin=0x7fc30401e0a0 dom=0x1ecb6b0 withResources=1 2013-09-17 08:50:50.198+0000: 17797: debug : virLockManagerNew:291 : plugin=0x7fc30401e0a0 type=0 nparams=4 params=0x7fc3103a3880 flags=0 2013-09-17 08:50:50.198+0000: 17797: debug : virLockManagerLogParams:98 : key=uuid type=uuid value=15f7c568-e477-43cd-20da-4b3830643b43 2013-09-17 08:50:50.198+0000: 17797: debug : virLockManagerLogParams:94 : key=name type=string value=FreeBSD 2013-09-17 08:50:50.198+0000: 17797: debug : virLockManagerLogParams:82 : key=id type=uint value=12 2013-09-17 08:50:50.198+0000: 17797: debug : virLockManagerLogParams:82 : key=pid type=uint value=17797 2013-09-17 08:50:50.198+0000: 17797: debug : virDomainLockManagerNew:135 : Adding leases 2013-09-17 08:50:50.198+0000: 17797: debug : virDomainLockManagerNew:140 : Adding disks 2013-09-17 08:50:50.198+0000: 17797: debug : virDomainLockManagerAddDisk:86 : Add disk /vmimages/gv0/FreeBSD.img 2013-09-17 08:50:50.198+0000: 17797: debug : virLockManagerAddResource:320 : lock=0x7fc3040c2b30 type=0 name=/vmimages/gv0/FreeBSD.img nparams=0 para 
ms=(nil) flags=0
2013-09-17 08:50:50.198+0000: 17797: debug : virLockManagerAcquire:337 : lock=0x7fc3040c2b30 state='(null)' flags=3 fd=0x7fc3103a3964 2013-09-17 08:50:50.198+0000: 17797: debug : virLockManagerFree:374 : lock=0x7fc3040c2b30 2013-09-17 08:50:50.198+0000: 17797: debug : qemuProcessHook:2585 : Moving procss to cgroup 2013-09-17 08:50:50.198+0000: 17797: debug : qemuProcessHook:2611 : Setup CPU affinity 2013-09-17 08:50:50.198+0000: 17797: debug : qemuProcessInitCpuAffinity:1808 : Setting CPU affinity 2013-09-17 08:50:50.199+0000: 17797: debug : qemuProcessInitCpuAffinity:1844 : Set CPU affinity with specified cpuset 2013-09-17 08:50:50.200+0000: 17797: debug : qemuProcessHook:2618 : Setting up security labelling 2013-09-17 08:50:50.200+0000: 17797: debug : virSecurityDACSetProcessLabel:642 : Dropping privileges of DEF to 105:111 2013-09-17 08:50:50.200+0000: 17797: debug : qemuProcessHook:2625 : Hook complete ret=0 2013-09-17 08:50:50.200+0000: 17797: debug : virCommandHook:2043 : Done hook 0 2013-09-17 08:50:50.200+0000: 17797: debug : virCommandHook:2056 : Notifying parent for handshake start on 33 2013-09-17 08:50:50.200+0000: 17797: debug : virCommandHook:2077 : Waiting on parent for handshake complete on 34 2013-09-17 08:50:50.208+0000: 17797: debug : virCommandHook:2093 : Hook is done 0 
char device redirected to /dev/pts/12
inet_listen_opts: bind(ipv4,0.0.0.0,49152): Address already in use
inet_listen_opts: FAILED
Migrate: Failed to bind socket
Migration failed. Exit code tcp:0.0.0.0:49152(-1), exiting.
2013-09-17 08:50:50.889+0000: shutting down
2013-09-17 08:50:54.727+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=31,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:11 -vga cirrus -incoming tcp:0.0.0.0:4 
9153 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
2013-09-17 08:50:54.733+0000: 17829: info : libvirt version: 0.9.12
2013-09-17 08:50:54.733+0000: 17829: debug : virCommandHook:2041 : Run hook 0x48e9e0 0x7fc3103a4320 2013-09-17 08:50:54.734+0000: 17829: debug : qemuProcessHook:2560 : Obtaining domain lock 2013-09-17 08:50:54.734+0000: 17829: debug : virDomainLockManagerNew:123 : plugin=0x7fc30401e0a0 dom=0x1ecb6b0 withResources=1 2013-09-17 08:50:54.734+0000: 17829: debug : virLockManagerNew:291 : plugin=0x7fc30401e0a0 type=0 nparams=4 params=0x7fc3103a3880 flags=0 2013-09-17 08:50:54.734+0000: 17829: debug : virLockManagerLogParams:98 : key=uuid type=uuid value=15f7c568-e477-43cd-20da-4b3830643b43 2013-09-17 08:50:54.734+0000: 17829: debug : virLockManagerLogParams:94 : key=name type=string value=FreeBSD 2013-09-17 08:50:54.734+0000: 17829: debug : virLockManagerLogParams:82 : key=id type=uint value=13 2013-09-17 08:50:54.734+0000: 17829: debug : virLockManagerLogParams:82 : key=pid type=uint value=17829 2013-09-17 08:50:54.734+0000: 17829: debug : virDomainLockManagerNew:135 : Adding leases 2013-09-17 08:50:54.734+0000: 17829: debug : virDomainLockManagerNew:140 : Adding disks 2013-09-17 08:50:54.734+0000: 17829: debug : virDomainLockManagerAddDisk:86 : Add disk /vmimages/gv0/FreeBSD.img 2013-09-17 08:50:54.734+0000: 17829: debug : virLockManagerAddResource:320 : lock=0x7fc3040f7850 type=0 name=/vmimages/gv0/FreeBSD.img nparams=0 para 
ms=(nil) flags=0
2013-09-17 08:50:54.734+0000: 17829: debug : virLockManagerAcquire:337 : lock=0x7fc3040f7850 state='(null)' flags=3 fd=0x7fc3103a3964 2013-09-17 08:50:54.734+0000: 17829: debug : virLockManagerFree:374 : lock=0x7fc3040f7850 2013-09-17 08:50:54.734+0000: 17829: debug : qemuProcessHook:2585 : Moving procss to cgroup 2013-09-17 08:50:54.734+0000: 17829: debug : qemuProcessHook:2611 : Setup CPU affinity 2013-09-17 08:50:54.734+0000: 17829: debug : qemuProcessInitCpuAffinity:1808 : Setting CPU affinity 2013-09-17 08:50:54.736+0000: 17829: debug : qemuProcessInitCpuAffinity:1844 : Set CPU affinity with specified cpuset 2013-09-17 08:50:54.736+0000: 17829: debug : qemuProcessHook:2618 : Setting up security labelling 2013-09-17 08:50:54.736+0000: 17829: debug : virSecurityDACSetProcessLabel:642 : Dropping privileges of DEF to 105:111 2013-09-17 08:50:54.737+0000: 17829: debug : qemuProcessHook:2625 : Hook complete ret=0 2013-09-17 08:50:54.737+0000: 17829: debug : virCommandHook:2043 : Done hook 0 2013-09-17 08:50:54.737+0000: 17829: debug : virCommandHook:2056 : Notifying parent for handshake start on 33 2013-09-17 08:50:54.737+0000: 17829: debug : virCommandHook:2077 : Waiting on parent for handshake complete on 34 2013-09-17 08:50:54.745+0000: 17829: debug : virCommandHook:2093 : Hook is done 0 
char device redirected to /dev/pts/12
2013-09-17 08:51:19.259+0000: shutting down
qemu: terminating on signal 15 from pid 17533
2013-09-17 08:51:33.083+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=31,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:11 -vga cirrus -incoming tcp:0.0.0.0:4 
9154 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
2013-09-17 08:51:33.090+0000: 17914: info : libvirt version: 0.9.12
2013-09-17 08:51:33.090+0000: 17914: debug : virCommandHook:2041 : Run hook 0x48e9e0 0x7fc30fba3320 2013-09-17 08:51:33.090+0000: 17914: debug : qemuProcessHook:2560 : Obtaining domain lock 2013-09-17 08:51:33.090+0000: 17914: debug : virDomainLockManagerNew:123 : plugin=0x7fc30401e0a0 dom=0x1ecb6b0 withResources=1 2013-09-17 08:51:33.090+0000: 17914: debug : virLockManagerNew:291 : plugin=0x7fc30401e0a0 type=0 nparams=4 params=0x7fc30fba2880 flags=0 2013-09-17 08:51:33.090+0000: 17914: debug : virLockManagerLogParams:98 : key=uuid type=uuid value=15f7c568-e477-43cd-20da-4b3830643b43 2013-09-17 08:51:33.090+0000: 17914: debug : virLockManagerLogParams:94 : key=name type=string value=FreeBSD 2013-09-17 08:51:33.090+0000: 17914: debug : virLockManagerLogParams:82 : key=id type=uint value=14 2013-09-17 08:51:33.090+0000: 17914: debug : virLockManagerLogParams:82 : key=pid type=uint value=17914 2013-09-17 08:51:33.090+0000: 17914: debug : virDomainLockManagerNew:135 : Adding leases 2013-09-17 08:51:33.090+0000: 17914: debug : virDomainLockManagerNew:140 : Adding disks 2013-09-17 08:51:33.090+0000: 17914: debug : virDomainLockManagerAddDisk:86 : Add disk /vmimages/gv0/FreeBSD.img 2013-09-17 08:51:33.090+0000: 17914: debug : virLockManagerAddResource:320 : lock=0x7fc3040c27b0 type=0 name=/vmimages/gv0/FreeBSD.img nparams=0 para 
ms=(nil) flags=0
2013-09-17 08:51:33.090+0000: 17914: debug : virLockManagerAcquire:337 : lock=0x7fc3040c27b0 state='(null)' flags=3 fd=0x7fc30fba2964 2013-09-17 08:51:33.090+0000: 17914: debug : virLockManagerFree:374 : lock=0x7fc3040c27b0 2013-09-17 08:51:33.091+0000: 17914: debug : qemuProcessHook:2585 : Moving procss to cgroup 2013-09-17 08:51:33.091+0000: 17914: debug : qemuProcessHook:2611 : Setup CPU affinity 2013-09-17 08:51:33.091+0000: 17914: debug : qemuProcessInitCpuAffinity:1808 : Setting CPU affinity 2013-09-17 08:51:33.093+0000: 17914: debug : qemuProcessInitCpuAffinity:1844 : Set CPU affinity with specified cpuset 2013-09-17 08:51:33.093+0000: 17914: debug : qemuProcessHook:2618 : Setting up security labelling 2013-09-17 08:51:33.093+0000: 17914: debug : virSecurityDACSetProcessLabel:642 : Dropping privileges of DEF to 105:111 2013-09-17 08:51:33.093+0000: 17914: debug : qemuProcessHook:2625 : Hook complete ret=0 2013-09-17 08:51:33.093+0000: 17914: debug : virCommandHook:2043 : Done hook 0 2013-09-17 08:51:33.093+0000: 17914: debug : virCommandHook:2056 : Notifying parent for handshake start on 33 2013-09-17 08:51:33.093+0000: 17914: debug : virCommandHook:2077 : Waiting on parent for handshake complete on 34 2013-09-17 08:51:33.102+0000: 17914: debug : virCommandHook:2093 : Hook is done 0 
char device redirected to /dev/pts/12
qemu: terminating on signal 15 from pid 17533
2013-09-17 08:53:04.210+0000: shutting down
2013-09-17 08:58:34.763+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=31,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:11 -vga cirrus -incoming tcp:0.0.0.0:4 
9152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
char device redirected to /dev/pts/12
inet_listen_opts: bind(ipv4,0.0.0.0,49152): Address already in use
inet_listen_opts: FAILED
Migrate: Failed to bind socket
Migration failed. Exit code tcp:0.0.0.0:49152(-1), exiting.
2013-09-17 08:58:35.415+0000: shutting down
2013-09-17 08:58:46.379+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-1.1 -cpu qemu64 -enable-kvm -m 5 12 -smp 1,sockets=1,cores=1,threads=1 -name FreeBSD -uuid 15f7c568-e477-43cd-20da-4b3830643b43 -no-user-config -nodefaults -chardev socket,id=charmon itor,path=/var/lib/libvirt/qemu/FreeBSD.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device pii x3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/vmimages/gv0/FreeBSD.img,if=none,id=drive-ide0-0-0,format=qcow2,cache=none -device ide-hd,bus= ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=31,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=52:54:00:36:1e:4c,b us=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:11 -vga cirrus -incoming tcp:0.0.0.0:4 
9153 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
char device redirected to /dev/pts/12
2013-09-17 08:59:08.925+0000: shutting down
qemu: terminating on signal 15 from pid 18134
reply via email to
[Prev in Thread] Current Thread [Next in Thread]