[Qemu-devel] Re: rtl8139 and qemu-0.11.0-rc2: NFS not responding

[Sven Rudolph]

Hello,

I originally reported this on the KVM list (address@hidden), but
I realized that the problem exists in the generic QEMU too.  So I
reproduced the problem with the non-KVM-specific QEMU versions (and
tried to bisect); hence I send you a changed/enhanced bug report:


With qemu-0.11.0-rc2 our diskless linux guests cannot use their NFS
root filesystem anymore.

  /usr/local/bin/qemu-system-x86_64 -m 4096 -smp 1 -boot n -net 
nic,macaddr=00:50:56:24:0b:57,model=rtl8139 -net 
tap,ifname=vm01,script=no,downscript=no

This boots via the pxe-rtl8139.bin Boot ROM and starts a
locally-developed diskless boot environment. (Unfortunately I'm not
able to describe this in detail, and I didn't manage to reproduce this
in an easier environment.) When this boot environment tries to mount
the new root filesystem via NFS, these message appear, waiting
severeal seconds between each line:

  nfs: server 172.31.11.10 not responding, still trying
  nfs: server 172.31.11.10 OK

This continues until I kill the qemu process. During this problem the
guests IP address can be pinged.

The problem exists in qemu-0.11.0-rc2, and it does not exist in qemu-0.10.6 .
I tried to bisect and reached this point:

  # git bisect good
  There are only 'skip'ped commit left to test.
  The first bad commit could be any of:
  6c042c16fc2453e147e8aed66510820302f50702
  8aeff62d75b72263d855a0b5892b8d52ad6f09e0
  e19eb22486f258a421108ac22b8380a4e2f16b97
  f10c592e8d35e59a11cf7af1484ab1051acc3ef6
  bbe2f399b222f1f2fcf5cd2ea78e4f5c9a66c64e
  f3b6c7fcf8fca857b3c3ba0aa5b3a06d7ce0ac37
  8fd2a2f1a9048b9e37a898c2a5e9ef59d0c1a095
  e94667b91ccfdb70164ae320b1c4ded6b5b8a3ec
  2d9aba3961dd6c18cdafecc8ce31b330b45e2723
  3e021d40b7548b2eeec62a082411c0745a5c635f
  015cb16699199b7c062f02a0b89a869fdb18330f
  4f1c942b7fb29864ad86cb3af9076da38f38f74e
  4ffb17f5c3244e405198ae285ffbb20a62e0d4b3
  e3f5ec2b5e92706e3b807059f79b1fb5d936e567
  cda9046ba7dbba45f3016e5d60caffa2d72960fa
  f8e76fbf5190575c0f927fe3c5b0ec6934c6c3fc
  068daedd7dffcd065d3f238a6c04bb2cf51a9cd2
  463af5349a787160642f023dad10eaf0cb419fb7
  df12c1f543b228daae7a2fbcfd3bb12a6faab38a
  We cannot bisect more!

(I git-skipped these commits because they didn't compile cleanly.)

My environment:
- Host
  - Linux 2.6.31-rc9 x86_64
    - kvm kernel components from this kernel
  - Dual-Socket AMD Opteron 2347 HE
- Guest
  - Linux 2.6.25.9 i686

I tried to watch this with tcpdump. Before the line 
"nfs: server 172.31.11.10 OK" it looks like this:

  13:45:28.665935 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.665940 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666162 IP 172.31.11.10.948098278 > 172.31.10.11.2049: 112 read [|nfs]
  13:45:28.666259 IP 172.31.11.10.964875494 > 172.31.10.11.2049: 112 read [|nfs]
  13:45:28.666345 IP 172.31.10.11.2049 > 172.31.11.10.948098278: reply ok 1472 
read
  13:45:28.666403 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666408 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666412 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666416 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666421 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666464 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666469 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666476 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666482 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666487 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666526 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666532 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666538 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666543 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666549 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666587 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666594 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666599 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666605 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666613 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666622 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666628 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666929 IP 172.31.10.11.2049 > 172.31.11.10.964875494: reply ok 1472 
read
  13:45:28.666935 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666940 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666944 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666949 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666991 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.666996 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667002 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667008 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667014 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667052 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667058 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667064 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667069 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667075 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667114 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667121 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667128 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667134 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667140 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667160 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667168 IP 172.31.10.11 > 172.31.11.10: udp
  13:45:28.667174 IP 172.31.10.11 > 172.31.11.10: udp

and after this line this appears:


  13:45:41.261362 IP 172.31.11.10 > 172.31.10.11: ICMP ip reassembly time 
exceeded, length 556
  13:45:41.682275 IP 172.31.11.10 > 172.31.10.11: ICMP ip reassembly time 
exceeded, length 556
  13:45:42.091219 IP 172.31.11.10 > 172.31.10.11: ICMP ip reassembly time 
exceeded, length 556
  13:45:42.942096 IP 172.31.11.10 > 172.31.10.11: ICMP ip reassembly time 
exceeded, length 556
  13:45:46.260603 arp who-has 172.31.10.11 tell 172.31.11.10
  13:45:46.260756 arp reply 172.31.10.11 is-at 00:c0:9f:ca:9a:78
  13:45:56.507139 IP 172.31.11.10 > 172.31.10.11: ICMP ip reassembly time 
exceeded, length 556
  13:45:57.207030 IP 172.31.11.10 > 172.31.10.11: ICMP ip reassembly time 
exceeded, length 556
  13:45:58.688814 IP 172.31.11.10 > 172.31.10.11: ICMP ip reassembly time 
exceeded, length 556


Thats all informatiion I extracted for now.

If needed I would try to provide additional information or run some
tests.

Thanks,

        Sven