qemu-discuss
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-discuss] qemu-ga causing soft lockup

From: Matthew Rahtz
Subject: [Qemu-discuss] qemu-ga causing soft lockup
Date: Sat, 16 Nov 2013 00:17:43 +0000 (GMT) 
Hello,

We have an Ubuntu 12.04.3 x86_64 guest running under KVM, on a host machine 
with the same setup. The backup process for the VM involves asking qemu-ga to 
freeze the guest's filesystems before an LVM snapshot is made of the volume 
that the VM resides on on the host.

A couple of nights ago, the script running the backup reported that it had had 
trouble unfreezing the volume after the snapshot was made - it only received 
the confirmation message back from qemu-ga over the socket after retrying some 
13 times over the course of about a minute (after which it read 12 confirmation 
messages all at once over the socket). Checking the kernel log, there were some 
worrying line looking like:

    BUG: soft lockup - CPU#0 stuck for 23s! [qemu-ga:1738]

repeated a few times.

Last night, the script ran again and the VM completely froze, without writing 
any log files indicating what the problem was (and unfortunately we were unable 
to capture the kernel panic over serial console). What we did see in the log 
file of the script itself though was a similar pattern as the previous night:

    Pinging guest through socket...
    Response: {"return": {}}
    Checking guest freeze status...
    Response: {"return": "thawed"}
    Freezing guest...
    !!!!!!!!!!!!! Ensuring guest is thawed...
    Error: didn't get a response from the socket, is qemu-ga running?
    Double checking guest freeze status...
    Ensuring guest is thawed...
    Double checking guest freeze status...
    Ensuring guest is thawed...
    Double checking guest freeze status...
    Ensuring guest is thawed...
    Double checking guest freeze status...
    Ensuring guest is thawed...
    Double checking guest freeze status...
    !!!!!!!!!!!!! Error: FAILED TO UNFREEZE GUEST

so the problem seemed to occur right at the point where the VM was frozen.

We have qemu-kvm 1.0+noroms-0ubuntu14.12 installed in the guest, and 
1.0+noroms-0ubuntu14.11 on the host.

We're still trying to figure out what triggered the complete lockup, so haven't 
tested to see whether it's reproducible yet. However, we have several other 
guests on the same host all with the same OS operating with no problems, albeit 
with a less intense workload. If anyone has any ideas what might be causing 
this, it would be of huge help. The full stack trace for the first failure is 
appended below. 

Thank you! :)


Nov 14 00:26:25 serverbox kernel: [297680.032034] BUG: soft lockup - CPU#0 
stuck for 23s! [qemu-ga:1738]
Nov 14 00:26:25 serverbox kernel: [297680.032034] Modules linked in: cirrus(F) 
ttm(F) drm_kms_helper(F) drm(F) parport_pc(F) ppdev(F) sysimgblt(F) psmouse(F) 
sysfillrect(F) virtio_balloon(F) microcode(F) syscopyarea(F) serio_raw(F) 
i2c_piix4(F) mac_hid(F) virtio_console(F) nfsd(F) nfs_acl(F) auth_rpcgss(F) 
nfs(F) fscache(F) lockd(F) lp(F) sunrpc(F) parport(F) ext2(F) floppy(F)
Nov 14 00:26:25 serverbox kernel: [297680.032034] CPU 0 
Nov 14 00:26:25 serverbox kernel: [297680.032034] Pid: 1738, comm: qemu-ga 
Tainted: GF       W    3.8.0-33-generic #48~precise1-Ubuntu Bochs Bochs
Nov 14 00:26:25 serverbox kernel: [297680.032034] RIP: 
0010:[<ffffffff811c4c3c>]  [<ffffffff811c4c3c>] wait_sb_inodes+0x7c/0x120
Nov 14 00:26:25 serverbox kernel: [297680.032034] RSP: 0018:ffff880118135d78  
EFLAGS: 00010206
Nov 14 00:26:25 serverbox kernel: [297680.032034] RAX: ffff880077dd3000 RBX: 
ffff880077dd4ef0 RCX: 000000000000001a
Nov 14 00:26:25 serverbox kernel: [297680.032034] RDX: 0000000000000006 RSI: 
ffff880118135c00 RDI: ffff880077dd3318
Nov 14 00:26:25 serverbox kernel: [297680.032034] RBP: ffff880118135da8 R08: 
0000000004000000 R09: 0000000000000001
Nov 14 00:26:25 serverbox kernel: [297680.032034] R10: 0000000000000100 R11: 
0000000000000001 R12: ffff880077dd4f78
Nov 14 00:26:25 serverbox kernel: [297680.032034] R13: ffff880077dd53c8 R14: 
ffff880115d9b4a0 R15: ffff880046df3620
Nov 14 00:26:25 serverbox kernel: [297680.032034] FS:  00007f03ce9ca700(0000) 
GS:ffff88011fc00000(0000) knlGS:0000000000000000
Nov 14 00:26:25 serverbox kernel: [297680.032034] CS:  0010 DS: 0000 ES: 0000 
CR0: 0000000080050033
Nov 14 00:26:25 serverbox kernel: [297680.032034] CR2: 00000000057c4000 CR3: 
0000000117963000 CR4: 00000000000006f0
Nov 14 00:26:25 serverbox kernel: [297680.032034] DR0: 0000000000000000 DR1: 
0000000000000000 DR2: 0000000000000000
Nov 14 00:26:25 serverbox kernel: [297680.032034] DR3: 0000000000000000 DR6: 
00000000ffff0ff0 DR7: 0000000000000400
Nov 14 00:26:25 serverbox kernel: [297680.032034] Process qemu-ga (pid: 1738, 
threadinfo ffff880118134000, task ffff880117952e80)
Nov 14 00:26:25 serverbox kernel: [297680.032034] Stack:
Nov 14 00:26:25 serverbox kernel: [297680.032034]  ffff880118135df8 
ffff880115d9b400 ffff880118135df8 ffff880118135db8
Nov 14 00:26:25 serverbox kernel: [297680.032034]  0000000000000000 
0000000000000000 ffff880118135e38 ffffffff811c4d7f
Nov 14 00:26:25 serverbox kernel: [297680.032034]  7fffffffffffffff 
ffff880115d9b400 ffff880117845d50 0000000000000001
Nov 14 00:26:25 serverbox kernel: [297680.032034] Call Trace:
Nov 14 00:26:25 serverbox kernel: [297680.032034]  [<ffffffff811c4d7f>] 
sync_inodes_sb+0x9f/0xd0
Nov 14 00:26:25 serverbox kernel: [297680.032034]  [<ffffffff811cc1cb>] 
__sync_filesystem+0x1b/0x60
Nov 14 00:26:25 serverbox kernel: [297680.032034]  [<ffffffff811cc28b>] 
sync_filesystem+0x4b/0x70
Nov 14 00:26:25 serverbox kernel: [297680.032034]  [<ffffffff8119e1b6>] 
freeze_super.part.8+0x66/0x100
Nov 14 00:26:25 serverbox kernel: [297680.032034]  [<ffffffff8119e29c>] 
freeze_super+0x4c/0x90
Nov 14 00:26:25 serverbox kernel: [297680.032034]  [<ffffffff811ad856>] 
do_vfs_ioctl+0x206/0x340
Nov 14 00:26:25 serverbox kernel: [297680.032034]  [<ffffffff811aaba9>] ? 
putname+0x29/0x40
Nov 14 00:26:25 serverbox kernel: [297680.032034]  [<ffffffff8119aff6>] ? 
do_sys_open+0x186/0x250
Nov 14 00:26:25 serverbox kernel: [297680.032034]  [<ffffffff811ada21>] 
sys_ioctl+0x91/0xb0
Nov 14 00:26:25 serverbox kernel: [297680.032034]  [<ffffffff816fd15d>] 
system_call_fastpath+0x1a/0x1f
Nov 14 00:26:25 serverbox kernel: [297680.032034] Code: 00 66 0f 1f 44 00 00 4c 
89 e7 e8 60 10 e8 ff 66 90 48 8b 83 00 01 00 00 49 39 c6 48 8d 98 00 ff ff ff 
74 6b 4c 8d a3 88 00 00 00 <4c> 8b 6b 30 4c 89 e7 e8 98 f6 52 00 f6 83 a0 00 00 
00 38 75 c7 
Nov 14 00:27:41 serverbox kernel: [297756.052018] BUG: soft lockup - CPU#1 
stuck for 22s! [qemu-ga:1738]
Nov 14 00:27:41 serverbox kernel: [297756.052018] Modules linked in: cirrus(F) 
ttm(F) drm_kms_helper(F) drm(F) parport_pc(F) ppdev(F) sysimgblt(F) psmouse(F) 
sysfillrect(F) virtio_balloon(F) microcode(F) syscopyarea(F) serio_raw(F) 
i2c_piix4(F) mac_hid(F) virtio_console(F) nfsd(F) nfs_acl(F) auth_rpcgss(F) 
nfs(F) fscache(F) lockd(F) lp(F) sunrpc(F) parport(F) ext2(F) floppy(F)
Nov 14 00:27:41 serverbox kernel: [297756.052018] CPU 1 
Nov 14 00:27:41 serverbox kernel: [297756.052018] Pid: 1738, comm: qemu-ga 
Tainted: GF       W    3.8.0-33-generic #48~precise1-Ubuntu Bochs Bochs
Nov 14 00:27:41 serverbox kernel: [297756.052018] RIP: 
0010:[<ffffffff811c4c3c>]  [<ffffffff811c4c3c>] wait_sb_inodes+0x7c/0x120
Nov 14 00:27:41 serverbox kernel: [297756.052018] RSP: 0018:ffff880118135d78  
EFLAGS: 00010202
Nov 14 00:27:41 serverbox kernel: [297756.052018] RAX: ffff8800bd0a9e30 RBX: 
ffff8800bd0add30 RCX: 000000000000001a
Nov 14 00:27:41 serverbox kernel: [297756.052018] RDX: 000000000000000b RSI: 
ffff880118135c00 RDI: ffff8800bd0aa148
Nov 14 00:27:41 serverbox kernel: [297756.052018] RBP: ffff880118135da8 R08: 
0000000004000000 R09: 0000000000000001
Nov 14 00:27:41 serverbox kernel: [297756.052018] R10: 0000000000000100 R11: 
0000000000000001 R12: ffff8800bd0addb8
Nov 14 00:27:41 serverbox kernel: [297756.052018] R13: ffff8800bd0ae208 R14: 
ffff880115d9b4a0 R15: ffff880069c4a0c0
Nov 14 00:27:41 serverbox kernel: [297756.052018] FS:  00007f03ce9ca700(0000) 
GS:ffff88011fd00000(0000) knlGS:0000000000000000
Nov 14 00:27:41 serverbox kernel: [297756.052018] CS:  0010 DS: 0000 ES: 0000 
CR0: 0000000080050033
Nov 14 00:27:41 serverbox kernel: [297756.052018] CR2: 000000000876d001 CR3: 
0000000117963000 CR4: 00000000000006e0
Nov 14 00:27:41 serverbox kernel: [297756.052018] DR0: 0000000000000000 DR1: 
0000000000000000 DR2: 0000000000000000
Nov 14 00:27:41 serverbox kernel: [297756.052018] DR3: 0000000000000000 DR6: 
00000000ffff0ff0 DR7: 0000000000000400
Nov 14 00:27:41 serverbox kernel: [297756.052018] Process qemu-ga (pid: 1738, 
threadinfo ffff880118134000, task ffff880117952e80)
Nov 14 00:27:41 serverbox kernel: [297756.052018] Stack:
Nov 14 00:27:41 serverbox kernel: [297756.052018]  ffff880118135df8 
ffff880115d9b400 ffff880118135df8 ffff880118135db8
Nov 14 00:27:41 serverbox kernel: [297756.052018]  0000000000000000 
0000000000000000 ffff880118135e38 ffffffff811c4d7f
Nov 14 00:27:41 serverbox kernel: [297756.052018]  7fffffffffffffff 
ffff880115d9b400 ffff880117845d50 0000000000000001
Nov 14 00:27:41 serverbox kernel: [297756.052018] Call Trace:
Nov 14 00:27:41 serverbox kernel: [297756.052018]  [<ffffffff811c4d7f>] 
sync_inodes_sb+0x9f/0xd0
Nov 14 00:27:41 serverbox kernel: [297756.052018]  [<ffffffff811cc1cb>] 
__sync_filesystem+0x1b/0x60
Nov 14 00:27:41 serverbox kernel: [297756.052018]  [<ffffffff811cc28b>] 
sync_filesystem+0x4b/0x70
Nov 14 00:27:41 serverbox kernel: [297756.052018]  [<ffffffff8119e1b6>] 
freeze_super.part.8+0x66/0x100
Nov 14 00:27:41 serverbox kernel: [297756.052018]  [<ffffffff8119e29c>] 
freeze_super+0x4c/0x90
Nov 14 00:27:41 serverbox kernel: [297756.052018]  [<ffffffff811ad856>] 
do_vfs_ioctl+0x206/0x340
Nov 14 00:27:41 serverbox kernel: [297756.052018]  [<ffffffff811aaba9>] ? 
putname+0x29/0x40
Nov 14 00:27:41 serverbox kernel: [297756.052018]  [<ffffffff8119aff6>] ? 
do_sys_open+0x186/0x250
Nov 14 00:27:41 serverbox kernel: [297756.052018]  [<ffffffff811ada21>] 
sys_ioctl+0x91/0xb0
Nov 14 00:27:41 serverbox kernel: [297756.052018]  [<ffffffff816fd15d>] 
system_call_fastpath+0x1a/0x1f
Nov 14 00:27:41 serverbox kernel: [297756.052018] Code: 00 66 0f 1f 44 00 00 4c 
89 e7 e8 60 10 e8 ff 66 90 48 8b 83 00 01 00 00 49 39 c6 48 8d 98 00 ff ff ff 
74 6b 4c 8d a3 88 00 00 00 <4c> 8b 6b 30 4c 89 e7 e8 98 f6 52 00 f6 83 a0 00 00 
00 38 75 c7 
Nov 14 00:28:09 serverbox kernel: [297784.052028] BUG: soft lockup - CPU#1 
stuck for 23s! [qemu-ga:1738]
Nov 14 00:28:09 serverbox kernel: [297784.052028] Modules linked in: cirrus(F) 
ttm(F) drm_kms_helper(F) drm(F) parport_pc(F) ppdev(F) sysimgblt(F) psmouse(F) 
sysfillrect(F) virtio_balloon(F) microcode(F) syscopyarea(F) serio_raw(F) 
i2c_piix4(F) mac_hid(F) virtio_console(F) nfsd(F) nfs_acl(F) auth_rpcgss(F) 
nfs(F) fscache(F) lockd(F) lp(F) sunrpc(F) parport(F) ext2(F) floppy(F)
Nov 14 00:28:09 serverbox kernel: [297784.052028] CPU 1 
Nov 14 00:28:09 serverbox kernel: [297784.052028] Pid: 1738, comm: qemu-ga 
Tainted: GF       W    3.8.0-33-generic #48~precise1-Ubuntu Bochs Bochs
Nov 14 00:28:09 serverbox kernel: [297784.052028] RIP: 
0010:[<ffffffff811c4c3c>]  [<ffffffff811c4c3c>] wait_sb_inodes+0x7c/0x120
Nov 14 00:28:09 serverbox kernel: [297784.052028] RSP: 0018:ffff880118135d78  
EFLAGS: 00010206
Nov 14 00:28:09 serverbox kernel: [297784.052028] RAX: ffff88003e643ab0 RBX: 
ffff88003e646f00 RCX: 000000000000001a
Nov 14 00:28:09 serverbox kernel: [297784.052028] RDX: 0000000000000010 RSI: 
ffff880118135c00 RDI: ffff88003e644138
Nov 14 00:28:09 serverbox kernel: [297784.052028] RBP: ffff880118135da8 R08: 
0000000004000000 R09: 0000000000000001
Nov 14 00:28:09 serverbox kernel: [297784.052028] R10: 0000000000000100 R11: 
0000000000000001 R12: ffff88003e646f88
Nov 14 00:28:09 serverbox kernel: [297784.052028] R13: ffff88003e6473d8 R14: 
ffff880115d9b4a0 R15: ffff880069c4a0c0
Nov 14 00:28:09 serverbox kernel: [297784.052028] FS:  00007f03ce9ca700(0000) 
GS:ffff88011fd00000(0000) knlGS:0000000000000000
Nov 14 00:28:09 serverbox kernel: [297784.052028] CS:  0010 DS: 0000 ES: 0000 
CR0: 0000000080050033
Nov 14 00:28:09 serverbox kernel: [297784.052028] CR2: 00000000096bf001 CR3: 
0000000117963000 CR4: 00000000000006e0
Nov 14 00:28:09 serverbox kernel: [297784.052028] DR0: 0000000000000000 DR1: 
0000000000000000 DR2: 0000000000000000
Nov 14 00:28:09 serverbox kernel: [297784.052028] DR3: 0000000000000000 DR6: 
00000000ffff0ff0 DR7: 0000000000000400
Nov 14 00:28:09 serverbox kernel: [297784.052028] Process qemu-ga (pid: 1738, 
threadinfo ffff880118134000, task ffff880117952e80)
Nov 14 00:28:09 serverbox kernel: [297784.052028] Stack:
Nov 14 00:28:09 serverbox kernel: [297784.052028]  ffff880118135df8 
ffff880115d9b400 ffff880118135df8 ffff880118135db8
Nov 14 00:28:09 serverbox kernel: [297784.052028]  0000000000000000 
0000000000000000 ffff880118135e38 ffffffff811c4d7f
Nov 14 00:28:09 serverbox kernel: [297784.052028]  7fffffffffffffff 
ffff880115d9b400 ffff880117845d50 0000000000000001
Nov 14 00:28:09 serverbox kernel: [297784.052028] Call Trace:
Nov 14 00:28:09 serverbox kernel: [297784.052028]  [<ffffffff811c4d7f>] 
sync_inodes_sb+0x9f/0xd0
Nov 14 00:28:09 serverbox kernel: [297784.052028]  [<ffffffff811cc1cb>] 
__sync_filesystem+0x1b/0x60
Nov 14 00:28:09 serverbox kernel: [297784.052028]  [<ffffffff811cc28b>] 
sync_filesystem+0x4b/0x70
Nov 14 00:28:09 serverbox kernel: [297784.052028]  [<ffffffff8119e1b6>] 
freeze_super.part.8+0x66/0x100
Nov 14 00:28:09 serverbox kernel: [297784.052028]  [<ffffffff8119e29c>] 
freeze_super+0x4c/0x90
Nov 14 00:28:09 serverbox kernel: [297784.052028]  [<ffffffff811ad856>] 
do_vfs_ioctl+0x206/0x340
Nov 14 00:28:09 serverbox kernel: [297784.052028]  [<ffffffff811aaba9>] ? 
putname+0x29/0x40
Nov 14 00:28:09 serverbox kernel: [297784.052028]  [<ffffffff8119aff6>] ? 
do_sys_open+0x186/0x250
Nov 14 00:28:09 serverbox kernel: [297784.052028]  [<ffffffff811ada21>] 
sys_ioctl+0x91/0xb0
Nov 14 00:28:09 serverbox kernel: [297784.052028]  [<ffffffff816fd15d>] 
system_call_fastpath+0x1a/0x1f
Nov 14 00:28:09 serverbox kernel: [297784.052028] Code: 00 66 0f 1f 44 00 00 4c 
89 e7 e8 60 10 e8 ff 66 90 48 8b 83 00 01 00 00 49 39 c6 48 8d 98 00 ff ff ff 
74 6b 4c 8d a3 88 00 00 00 <4c> 8b 6b 30 4c 89 e7 e8 98 f6 52 00 f6 83 a0 00 00 
00 38 75 c7 
Nov 14 00:28:17 serverbox kernel: [297791.720022] INFO: rcu_sched self-detected 
stall on CPU { 1}  (t=15000 jiffies g=5187133 c=5187132 q=712)
Nov 14 00:28:17 serverbox kernel: [297791.720022] sending NMI to all CPUs:
Nov 14 00:28:17 serverbox kernel: [297791.720022] NMI backtrace for cpu 1
Nov 14 00:28:17 serverbox kernel: [297791.720022] CPU 1 
Nov 14 00:28:17 serverbox kernel: [297791.720022] Pid: 1738, comm: qemu-ga 
Tainted: GF       W    3.8.0-33-generic #48~precise1-Ubuntu Bochs Bochs
Nov 14 00:28:17 serverbox kernel: [297791.720022] RIP: 
0010:[<ffffffff8103efff>]  [<ffffffff8103efff>] flat_send_IPI_all+0xaf/0xd0
Nov 14 00:28:17 serverbox kernel: [297791.720022] RSP: 0018:ffff88011fd03d38  
EFLAGS: 00010006
Nov 14 00:28:17 serverbox kernel: [297791.720022] RAX: 0000000000000000 RBX: 
0000000000000046 RCX: ffffffff81e314c0
Nov 14 00:28:17 serverbox kernel: [297791.720022] RDX: 0000000000004279 RSI: 
0000000000000086 RDI: 0000000000000300
Nov 14 00:28:17 serverbox kernel: [297791.720022] RBP: ffff88011fd03d58 R08: 
000000000000000a R09: 0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.720022] R10: 0000000000000245 R11: 
0000000000000244 R12: 0000000000000c00
Nov 14 00:28:17 serverbox kernel: [297791.720022] R13: 0000000003000000 R14: 
0000000000000001 R15: 0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.720022] FS:  00007f03ce9ca700(0000) 
GS:ffff88011fd00000(0000) knlGS:0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.720022] CS:  0010 DS: 0000 ES: 0000 
CR0: 0000000080050033
Nov 14 00:28:17 serverbox kernel: [297791.720022] CR2: 0000000009bc8001 CR3: 
0000000117963000 CR4: 00000000000006e0
Nov 14 00:28:17 serverbox kernel: [297791.720022] DR0: 0000000000000000 DR1: 
0000000000000000 DR2: 0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.720022] DR3: 0000000000000000 DR6: 
00000000ffff0ff0 DR7: 0000000000000400
Nov 14 00:28:17 serverbox kernel: [297791.720022] Process qemu-ga (pid: 1738, 
threadinfo ffff880118134000, task ffff880117952e80)
Nov 14 00:28:17 serverbox kernel: [297791.720022] Stack:
Nov 14 00:28:17 serverbox kernel: [297791.720022]  0000000000004274 
0000000000002710 00000000000002c8 ffffffff81ce7080
Nov 14 00:28:17 serverbox kernel: [297791.720022]  ffff88011fd03d78 
ffffffff8103ab01 00000000000002c8 ffffffff81c3f440
Nov 14 00:28:17 serverbox kernel: [297791.720022]  ffff88011fd03da8 
ffffffff810f65c4 ffff88011fd03dc8 ffffffff81c3f440
Nov 14 00:28:17 serverbox kernel: [297791.720022] Call Trace:
Nov 14 00:28:17 serverbox kernel: [297791.720022]  <IRQ> 
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff8103ab01>] 
arch_trigger_all_cpu_backtrace+0x61/0xa0
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff810f65c4>] 
print_cpu_stall+0xd4/0x180
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff810f66e4>] 
__rcu_pending+0x74/0x210
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff810f68dd>] 
rcu_pending+0x5d/0x80
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff810f86c3>] 
rcu_check_callbacks+0xa3/0x160
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff8106c038>] 
update_process_times+0x48/0x90
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff810b6ba3>] 
tick_sched_handle.isra.8+0x33/0x70
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff810b6ccc>] 
tick_sched_timer+0x4c/0x80
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff810839d8>] 
__run_hrtimer+0x78/0x1f0
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff810b6c80>] ? 
tick_nohz_handler+0xa0/0xa0
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff810b002c>] ? 
ktime_get_update_offsets+0x4c/0xd0
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff81084267>] 
hrtimer_interrupt+0xf7/0x240
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff816fe5dc>] ? 
call_softirq+0x1c/0x30
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff816fef59>] 
smp_apic_timer_interrupt+0x69/0x99
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff816fde9d>] 
apic_timer_interrupt+0x6d/0x80
Nov 14 00:28:17 serverbox kernel: [297791.720022]  <EOI> 
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff811c4c3c>] ? 
wait_sb_inodes+0x7c/0x120
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff811c4c20>] ? 
wait_sb_inodes+0x60/0x120
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff811c4d7f>] 
sync_inodes_sb+0x9f/0xd0
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff811cc1cb>] 
__sync_filesystem+0x1b/0x60
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff811cc28b>] 
sync_filesystem+0x4b/0x70
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff8119e1b6>] 
freeze_super.part.8+0x66/0x100
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff8119e29c>] 
freeze_super+0x4c/0x90
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff811ad856>] 
do_vfs_ioctl+0x206/0x340
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff811aaba9>] ? 
putname+0x29/0x40
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff8119aff6>] ? 
do_sys_open+0x186/0x250
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff811ada21>] 
sys_ioctl+0x91/0xb0
Nov 14 00:28:17 serverbox kernel: [297791.720022]  [<ffffffff816fd15d>] 
system_call_fastpath+0x1a/0x1f
Nov 14 00:28:17 serverbox kernel: [297791.720022] Code: 8b 05 06 7d ca 00 41 c1 
e5 18 44 8b 60 3c ff 90 50 01 00 00 44 89 2c 25 10 73 5f ff 41 81 cc 00 04 00 
00 44 89 24 25 00 73 5f ff <48> 89 df 57 9d 0f 1f 44 00 00 48 83 c4 08 5b 41 5c 
41 5d 5d c3 
Nov 14 00:28:17 serverbox kernel: [297791.720225] NMI backtrace for cpu 0
Nov 14 00:28:17 serverbox kernel: [297791.720225] CPU 0 
Nov 14 00:28:17 serverbox kernel: [297791.720225] Pid: 0, comm: swapper/0 
Tainted: GF       W    3.8.0-33-generic #48~precise1-Ubuntu Bochs Bochs
Nov 14 00:28:17 serverbox kernel: [297791.720225] RIP: 
0010:[<ffffffff81045136>]  [<ffffffff81045136>] native_safe_halt+0x6/0x10
Nov 14 00:28:17 serverbox kernel: [297791.720225] RSP: 0018:ffffffff81c01e98  
EFLAGS: 00000246
Nov 14 00:28:17 serverbox kernel: [297791.720225] RAX: 0000000000000000 RBX: 
ffffffff81c01fd8 RCX: 00000001046ed73c
Nov 14 00:28:17 serverbox kernel: [297791.720225] RDX: ffffffff81e52968 RSI: 
0000000000000082 RDI: 0000000000000092
Nov 14 00:28:17 serverbox kernel: [297791.720225] RBP: ffffffff81c01e98 R08: 
0000000000000000 R09: 0000000000000001
Nov 14 00:28:17 serverbox kernel: [297791.720225] R10: 0000000000000000 R11: 
0000000000000000 R12: ffffffff81ce7060
Nov 14 00:28:17 serverbox kernel: [297791.720225] R13: 0000000000000000 R14: 
ffff88011ffac780 R15: 000000000008c800
Nov 14 00:28:17 serverbox kernel: [297791.720225] FS:  0000000000000000(0000) 
GS:ffff88011fc00000(0000) knlGS:0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.720225] CS:  0010 DS: 0000 ES: 0000 
CR0: 000000008005003b
Nov 14 00:28:17 serverbox kernel: [297791.720225] CR2: 00007fd4db1b4000 CR3: 
0000000118018000 CR4: 00000000000006f0
Nov 14 00:28:17 serverbox kernel: [297791.720225] DR0: 0000000000000000 DR1: 
0000000000000000 DR2: 0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.720225] DR3: 0000000000000000 DR6: 
00000000ffff0ff0 DR7: 0000000000000400
Nov 14 00:28:17 serverbox kernel: [297791.720225] Process swapper/0 (pid: 0, 
threadinfo ffffffff81c00000, task ffffffff81c15440)
Nov 14 00:28:17 serverbox kernel: [297791.720225] Stack:
Nov 14 00:28:17 serverbox kernel: [297791.720225]  ffffffff81c01ec8 
ffffffff8101cc73 ffffffff81c01fd8 ffffffff81ce7060
Nov 14 00:28:17 serverbox kernel: [297791.720225]  ffffffff81d592e0 
ffff88011ffac780 ffffffff81c01ef8 ffffffff8101db19
Nov 14 00:28:17 serverbox kernel: [297791.720225]  ffffffff81c01ee8 
144fa9e4346f1e9c 0000000000000000 ffffffff81d4f000
Nov 14 00:28:17 serverbox kernel: [297791.720225] Call Trace:
Nov 14 00:28:17 serverbox kernel: [297791.720225]  [<ffffffff8101cc73>] 
default_idle+0x53/0x1f0
Nov 14 00:28:17 serverbox kernel: [297791.720225]  [<ffffffff8101db19>] 
cpu_idle+0xd9/0x120
Nov 14 00:28:17 serverbox kernel: [297791.720225]  [<ffffffff816c76f2>] 
rest_init+0x72/0x80
Nov 14 00:28:17 serverbox kernel: [297791.720225]  [<ffffffff81d05c4f>] 
start_kernel+0x3d1/0x3de
Nov 14 00:28:17 serverbox kernel: [297791.720225]  [<ffffffff81d057ff>] ? 
pass_bootoption.constprop.2+0xd3/0xd3
Nov 14 00:28:17 serverbox kernel: [297791.720225]  [<ffffffff81d05397>] 
x86_64_start_reservations+0x131/0x135
Nov 14 00:28:17 serverbox kernel: [297791.720225]  [<ffffffff81d05120>] ? 
early_idt_handlers+0x120/0x120
Nov 14 00:28:17 serverbox kernel: [297791.720225]  [<ffffffff81d05468>] 
x86_64_start_kernel+0xcd/0xdc
Nov 14 00:28:17 serverbox kernel: [297791.720225] Code: 00 00 00 00 00 55 48 89 
e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 
00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 
66 0f 1f 84 
Nov 14 00:28:17 serverbox kernel: [297791.765057] INFO: rcu_sched detected 
stalls on CPUs/tasks: { 1} (detected by 0, t=15011 jiffies, g=5187133, 
c=5187132, q=712)
Nov 14 00:28:17 serverbox kernel: [297791.765730] sending NMI to all CPUs:
Nov 14 00:28:17 serverbox kernel: [297791.765740] NMI backtrace for cpu 0
Nov 14 00:28:17 serverbox kernel: [297791.765741] CPU 0 
Nov 14 00:28:17 serverbox kernel: [297791.765752] Pid: 0, comm: swapper/0 
Tainted: GF       W    3.8.0-33-generic #48~precise1-Ubuntu Bochs Bochs
Nov 14 00:28:17 serverbox kernel: [297791.765753] RIP: 
0010:[<ffffffff8103efff>]  [<ffffffff8103efff>] flat_send_IPI_all+0xaf/0xd0
Nov 14 00:28:17 serverbox kernel: [297791.765755] RSP: 0018:ffff88011fc03d28  
EFLAGS: 00010006
Nov 14 00:28:17 serverbox kernel: [297791.765756] RAX: 0000000000000000 RBX: 
0000000000000046 RCX: ffffffff81e314c0
Nov 14 00:28:17 serverbox kernel: [297791.765757] RDX: 0000000000004717 RSI: 
0000000000000082 RDI: 0000000000000300
Nov 14 00:28:17 serverbox kernel: [297791.765758] RBP: ffff88011fc03d48 R08: 
000000000000000a R09: 0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.765759] R10: 000000000000029a R11: 
0000000000000299 R12: 0000000000000c00
Nov 14 00:28:17 serverbox kernel: [297791.765760] R13: 0000000003000000 R14: 
ffffffff81ce7080 R15: 0000000000000082
Nov 14 00:28:17 serverbox kernel: [297791.765762] FS:  0000000000000000(0000) 
GS:ffff88011fc00000(0000) knlGS:0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.765763] CS:  0010 DS: 0000 ES: 0000 
CR0: 000000008005003b
Nov 14 00:28:17 serverbox kernel: [297791.765764] CR2: 00007fd4db1b4000 CR3: 
0000000118018000 CR4: 00000000000006f0
Nov 14 00:28:17 serverbox kernel: [297791.765767] DR0: 0000000000000000 DR1: 
0000000000000000 DR2: 0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.765770] DR3: 0000000000000000 DR6: 
00000000ffff0ff0 DR7: 0000000000000400
Nov 14 00:28:17 serverbox kernel: [297791.765771] Process swapper/0 (pid: 0, 
threadinfo ffffffff81c00000, task ffffffff81c15440)
Nov 14 00:28:17 serverbox kernel: [297791.765772] Stack:
Nov 14 00:28:17 serverbox kernel: [297791.765772]  0000000000004712 
0000000000002710 0000000000000001 00000000000002c8
Nov 14 00:28:17 serverbox kernel: [297791.765774]  ffff88011fc03d68 
ffffffff8103ab01 00000000004f263c ffffffff81c3f440
Nov 14 00:28:17 serverbox kernel: [297791.765776]  ffff88011fc03da8 
ffffffff810f5b8b ffff88011fc03da8 ffffffff81c3f440
Nov 14 00:28:17 serverbox kernel: [297791.765778] Call Trace:
Nov 14 00:28:17 serverbox kernel: [297791.765779]  <IRQ> 
Nov 14 00:28:17 serverbox kernel: [297791.765780]  [<ffffffff8103ab01>] 
arch_trigger_all_cpu_backtrace+0x61/0xa0
Nov 14 00:28:17 serverbox kernel: [297791.765785]  [<ffffffff810f5b8b>] 
print_other_cpu_stall+0x21b/0x280
Nov 14 00:28:17 serverbox kernel: [297791.765787]  [<ffffffff810f6810>] 
__rcu_pending+0x1a0/0x210
Nov 14 00:28:17 serverbox kernel: [297791.765789]  [<ffffffff810f68dd>] 
rcu_pending+0x5d/0x80
Nov 14 00:28:17 serverbox kernel: [297791.765791]  [<ffffffff810f86c3>] 
rcu_check_callbacks+0xa3/0x160
Nov 14 00:28:17 serverbox kernel: [297791.765794]  [<ffffffff8106c038>] 
update_process_times+0x48/0x90
Nov 14 00:28:17 serverbox kernel: [297791.765796]  [<ffffffff810b6ba3>] 
tick_sched_handle.isra.8+0x33/0x70
Nov 14 00:28:17 serverbox kernel: [297791.765798]  [<ffffffff810b6ccc>] 
tick_sched_timer+0x4c/0x80
Nov 14 00:28:17 serverbox kernel: [297791.765808]  [<ffffffff810839d8>] 
__run_hrtimer+0x78/0x1f0
Nov 14 00:28:17 serverbox kernel: [297791.765811]  [<ffffffff810b6c80>] ? 
tick_nohz_handler+0xa0/0xa0
Nov 14 00:28:17 serverbox kernel: [297791.765813]  [<ffffffff810b002c>] ? 
ktime_get_update_offsets+0x4c/0xd0
Nov 14 00:28:17 serverbox kernel: [297791.765815]  [<ffffffff81084267>] 
hrtimer_interrupt+0xf7/0x240
Nov 14 00:28:17 serverbox kernel: [297791.765818]  [<ffffffff816fef59>] 
smp_apic_timer_interrupt+0x69/0x99
Nov 14 00:28:17 serverbox kernel: [297791.765820]  [<ffffffff816fde9d>] 
apic_timer_interrupt+0x6d/0x80
Nov 14 00:28:17 serverbox kernel: [297791.765821]  <EOI> 
Nov 14 00:28:17 serverbox kernel: [297791.765822]  [<ffffffff81045136>] ? 
native_safe_halt+0x6/0x10
Nov 14 00:28:17 serverbox kernel: [297791.765826]  [<ffffffff8101cc73>] 
default_idle+0x53/0x1f0
Nov 14 00:28:17 serverbox kernel: [297791.765828]  [<ffffffff8101db19>] 
cpu_idle+0xd9/0x120
Nov 14 00:28:17 serverbox kernel: [297791.765831]  [<ffffffff816c76f2>] 
rest_init+0x72/0x80
Nov 14 00:28:17 serverbox kernel: [297791.765833]  [<ffffffff81d05c4f>] 
start_kernel+0x3d1/0x3de
Nov 14 00:28:17 serverbox kernel: [297791.765835]  [<ffffffff81d057ff>] ? 
pass_bootoption.constprop.2+0xd3/0xd3
Nov 14 00:28:17 serverbox kernel: [297791.765837]  [<ffffffff81d05397>] 
x86_64_start_reservations+0x131/0x135
Nov 14 00:28:17 serverbox kernel: [297791.765839]  [<ffffffff81d05120>] ? 
early_idt_handlers+0x120/0x120
Nov 14 00:28:17 serverbox kernel: [297791.765841]  [<ffffffff81d05468>] 
x86_64_start_kernel+0xcd/0xdc
Nov 14 00:28:17 serverbox kernel: [297791.765842] Code: 8b 05 06 7d ca 00 41 c1 
e5 18 44 8b 60 3c ff 90 50 01 00 00 44 89 2c 25 10 73 5f ff 41 81 cc 00 04 00 
00 44 89 24 25 00 73 5f ff <48> 89 df 57 9d 0f 1f 44 00 00 48 83 c4 08 5b 41 5c 
41 5d 5d c3 
Nov 14 00:28:17 serverbox kernel: [297791.765861] NMI backtrace for cpu 1
Nov 14 00:28:17 serverbox kernel: [297791.765863] CPU 1 
Nov 14 00:28:17 serverbox kernel: [297791.765865] Pid: 1738, comm: qemu-ga 
Tainted: GF       W    3.8.0-33-generic #48~precise1-Ubuntu Bochs Bochs
Nov 14 00:28:17 serverbox kernel: [297791.765867] RIP: 
0010:[<ffffffff811c4c3c>]  [<ffffffff811c4c3c>] wait_sb_inodes+0x7c/0x120
Nov 14 00:28:17 serverbox kernel: [297791.765870] RSP: 0018:ffff880118135d78  
EFLAGS: 00010206
Nov 14 00:28:17 serverbox kernel: [297791.765872] RAX: ffff88006cc6f000 RBX: 
ffff88006cc6ef00 RCX: 000000000000001a
Nov 14 00:28:17 serverbox kernel: [297791.765873] RDX: 000000000000000d RSI: 
ffff880118135c00 RDI: ffff88006cc6f318
Nov 14 00:28:17 serverbox kernel: [297791.765873] RBP: ffff880118135da8 R08: 
0000000004000000 R09: 0000000000000001
Nov 14 00:28:17 serverbox kernel: [297791.765874] R10: 0000000000000100 R11: 
0000000000000001 R12: ffff88006cc6ef88
Nov 14 00:28:17 serverbox kernel: [297791.765875] R13: ffff88006cc6f3d8 R14: 
ffff880115d9b4a0 R15: ffff880069c4a0c0
Nov 14 00:28:17 serverbox kernel: [297791.765877] FS:  00007f03ce9ca700(0000) 
GS:ffff88011fd00000(0000) knlGS:0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.765878] CS:  0010 DS: 0000 ES: 0000 
CR0: 0000000080050033
Nov 14 00:28:17 serverbox kernel: [297791.765879] CR2: 0000000009bca001 CR3: 
0000000117963000 CR4: 00000000000006e0
Nov 14 00:28:17 serverbox kernel: [297791.765882] DR0: 0000000000000000 DR1: 
0000000000000000 DR2: 0000000000000000
Nov 14 00:28:17 serverbox kernel: [297791.765885] DR3: 0000000000000000 DR6: 
00000000ffff0ff0 DR7: 0000000000000400
Nov 14 00:28:17 serverbox kernel: [297791.765886] Process qemu-ga (pid: 1738, 
threadinfo ffff880118134000, task ffff880117952e80)
Nov 14 00:28:17 serverbox kernel: [297791.765887] Stack:
Nov 14 00:28:17 serverbox kernel: [297791.765888]  ffff880118135df8 
ffff880115d9b400 ffff880118135df8 ffff880118135db8
Nov 14 00:28:17 serverbox kernel: [297791.765890]  0000000000000000 
0000000000000000 ffff880118135e38 ffffffff811c4d7f
Nov 14 00:28:17 serverbox kernel: [297791.765892]  7fffffffffffffff 
ffff880115d9b400 ffff880117845d50 0000000000000001
Nov 14 00:28:17 serverbox kernel: [297791.765893] Call Trace:
Nov 14 00:28:17 serverbox kernel: [297791.765896]  [<ffffffff811c4d7f>] 
sync_inodes_sb+0x9f/0xd0
Nov 14 00:28:17 serverbox kernel: [297791.765898]  [<ffffffff811cc1cb>] 
__sync_filesystem+0x1b/0x60
Nov 14 00:28:17 serverbox kernel: [297791.765900]  [<ffffffff811cc28b>] 
sync_filesystem+0x4b/0x70
Nov 14 00:28:17 serverbox kernel: [297791.765902]  [<ffffffff8119e1b6>] 
freeze_super.part.8+0x66/0x100
Nov 14 00:28:17 serverbox kernel: [297791.765904]  [<ffffffff8119e29c>] 
freeze_super+0x4c/0x90
Nov 14 00:28:17 serverbox kernel: [297791.765906]  [<ffffffff811ad856>] 
do_vfs_ioctl+0x206/0x340
Nov 14 00:28:17 serverbox kernel: [297791.765908]  [<ffffffff811aaba9>] ? 
putname+0x29/0x40
Nov 14 00:28:17 serverbox kernel: [297791.765910]  [<ffffffff8119aff6>] ? 
do_sys_open+0x186/0x250
Nov 14 00:28:17 serverbox kernel: [297791.765912]  [<ffffffff811ada21>] 
sys_ioctl+0x91/0xb0
Nov 14 00:28:17 serverbox kernel: [297791.765914]  [<ffffffff816fd15d>] 
system_call_fastpath+0x1a/0x1f
Nov 14 00:28:17 serverbox kernel: [297791.765915] Code: 00 66 0f 1f 44 00 00 4c 
89 e7 e8 60 10 e8 ff 66 90 48 8b 83 00 01 00 00 49 39 c6 48 8d 98 00 ff ff ff 
74 6b 4c 8d a3 88 00 00 00 <4c> 8b 6b 30 4c 89 e7 e8 98 f6 52 00 f6 83 a0 00 00 
00 38 75 c7 


Stay informed by joining the Rapita Systems mailing list
http://www.rapitasystems.com/rapita/mailing_list

For real-time verifications issues and discussion, follow
the Rapita Systems blog
http://www.rapitasystems.com/blog
reply via email to
[Prev in Thread] Current Thread [Next in Thread]