RE: [Bug-wget] [wget => Kernel message] Too many open files

[Minato Namikaze]

> From: address@hidden
> To: address@hidden
> CC: address@hidden
> Subject: Re: [Bug-wget] [wget => Kernel message] Too many open files
> Date: Thu, 15 Jul 2010 13:51:40 +0200
> 
> Hi Minato,
> 
> I wasn't able to reproduce this problem, I have tried with a directory
> containing 50000 files.
> 

I have to precise that the 500,000 files were downloaded by wget, thus are 
checked for existance every times I launch the command I described in my 
previous mail.

> 
> What do you get using this command (be sure that website.com is the real
> one)?
> 
> strace -e open,socket wget -q -nc -r -l inf --no-remove-listing \
>        http://website.com/ 2>&1 | tail
> 

I launched this command, but I won't be able to tell you the result before the 
error occurs. I won't send the mail until then.
Meanwhile, I learned with free -m that wget uses approximately 1.8 GB (about 
2.4 GB when the second one was running) of my RAM memory as a buffer and I 
(only?) have 4 GB RAM.
The result of your command is : 
open("www.website.com/folders/index.html", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 
EMFILE (Too many open files)
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 1023
for all files after ~14 min of consenting download (after over 2hours checking 
the 500k files) between wget and strace.
I didn't use the 'tail' so I could monitor what happened.
I made a Scroll lock to pause it, and when I relaunched it, it noticed the 
files that have been downloaded by the other wget, until it arrived at a 
collision (again) and then the error messages changed to :
socket(PF_NETLINK, SOCK_RAW, 0)         = -1 EMFILE (Too many open 
files)

open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = -1 EMFILE (Too many open files)
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = -1 EMFILE (Too many 
open files)
socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = -1 EMFILE (Too many 
open files)
.
For both it is a very fast repetitive message, so maybe I made a circulary 
mistake.
I let this error continue until it reached the end of the mirror-planned list.
The tail is :
socket(PF_NETLINK, SOCK_RAW, 0)         = -1 EMFILE (Too many open 
files)


open("/etc/hosts", O_RDONLY|O_CLOEXEC)  = -1 EMFILE (Too many open 
files)

socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = -1 EMFILE (Too 
many open files)

socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = -1 EMFILE (Too 
many open files)

(so I didn't make a circular mistake on this one...)

> 
> Can you check your "ulimit -n"?
> 

Result : 1024
btw, I run wget as root, and strace as well.

> 
> Cheers,
> Giuseppe
> 

I forgot to tell that the destination is a NTFS fs mounted with ntfs-3g. Should 
I try to copy the files on a ext4 and retry to test if the problem comes from 
there ?


Thank you for you answer
And sorry for the long message...
Minato
> 
> 
> Minato Namikaze <address@hidden> writes:
> 
> > First of all : sorry for displaying the french version of wget log messages.
> >
> > Bug (?) type : Kernel message
> > Reproduce : wget -nc -r -l inf --no-remove-listing http://website.com/
> > (launched TWICE in different terminals, and with a lot of files already 
> > downloaded)
> >
> > Explanation : The two instances will pass over the numerous files already 
> > downloaded. After that, they will try to download the first not downloaded 
> > file.
> > Then, one of thoses cases appear :
> > Case 1 : one will be faster and they will download one file out of two each 
> > (during approximatively 10~20 minutes max until case 2 appears)
> > Case 2 : one will refuse to download and will display the next error 
> > message for all files.
> >
> > Log :
> > --2010-07-13 14:04:35--  http://www.website.com/folders/
> > Connexion vers site|ip|:80...connecté.
> > requête HTTP transmise, en attente de la réponse...200 OK
> > Longueur: 11886 (12K) [text/html]
> > www.website.com/folders/file.html: Too many open files
> >
> > Impossible d'écrire dans <<www.website.com/folders/file.html>> (Too many 
> > open files).
> > /endlog
> >
> >
> > This is the message displayed for all files by ONE of the wgets running
> > Information: if I stop the failing one, relaunch it, and wait until all 
> > files have been not-clobbered, I have great chances that the other one will 
> > fail with the same message(s).
> >
> > Information: /proc/sys/fs/file-max is set at 405540.
> > Information: the folder where the website is mirrored contains currently 
> > 525000 html files and as much folders (total: 6.6 GB).
> > Information: When I have a (BIG) network problem, and whet cancels every 
> > single file because the address can't be resolved, the elapsed time 
> > indicated is always 0s. (and thus the average speed is enormous)
> >
> >
> > If this mailing list was not the place where I should have posted this, 
> > please let me know.
> >                                       
> > _________________________________________________________________
> > Vous voulez regarder la TV directement depuis votre PC ? C'est très simple 
> > avec Windows 7
> > http://clk.atdmt.com/FRM/go/229960614/direct/01/
                                          
_________________________________________________________________
Allumez et éteignez votre PC en un instant avec Windows 7 !
http://clk.atdmt.com/FRM/go/238030931/direct/01/