[Bug-wget] robots.txt seemingly ignored

[Daniel Feenberg]

I have the following wget command line:

   wget -r  http://wwwdev.nber.org/

http://wwwdev.nber.org/robots.txt  is:

  User-agent: *
  Disallow: /

  User-Agent: W3C-checklink
  Disallow:


However wget fetches thousands of pages from wwwdev.nber.org. I would have 
thought nothing would be found. (This is a demonstration, obviously in 
real life I'd have a more detailed robots.txt to control the process).

Obviously too, I don't understand something about wget or robots.txt. Can 
anyone help me out?

This is GNU Wget 1.12 built on linux-gnu.

Thank you
Daniel Feenberg