[Bug-wget] parallel wget...

[Giuseppe Scrivano]

hello,

The winter is coming, not much to do outside and I have spent the day
working on something I had in mind already for too long.

Unfortunately I couldn't start the implementation as I have thought it
could be possible, there are too many nested `select' points in the code
and implement an event-driven single-thread parallel wget seems like too
much work.

I have used different threads, spawned by retrieve_tree in the recur.c,
I haven't published the code yet[1] since it is just a ugly hack for now
and still sometimes it segfaults, it will take a while before I can go
trough the code and ensure it is reentrant and can be used by different
threads without problems.

But I would like to share some results with you....

$ LANG=C wget --version | head -n 1
GNU Wget 1.13 built on linux-gnu.

$ LANG=C ./wget --version | head -n 1
GNU Wget 1.13.4-2567-dirty built on linux-gnu.

$ rm -rf it.gnu.org/ && time wget -q --no-http-keep-alive -r -np 
http://it.gnu.org/~gscrivano/files/parallel/

real    0m2.808s
user    0m0.008s
sys     0m0.020s
$ rm -rf it.gnu.org/ && time ./wget --jobs=2 -q --no-http-keep-alive -r -np 
http://it.gnu.org/~gscrivano/files/parallel/

real    0m1.291s
user    0m0.004s
sys     0m0.016s
$ rm -rf it.gnu.org/ && time ./wget --jobs=4 -q --no-http-keep-alive -r -np 
http://it.gnu.org/~gscrivano/files/parallel/

real    0m0.521s
user    0m0.008s
sys     0m0.012s
$ rm -rf it.gnu.org/ && time ./wget --jobs=8 -q --no-http-keep-alive -r -np 
http://it.gnu.org/~gscrivano/files/parallel/

real    0m0.395s
user    0m0.008s
sys     0m0.004s


Nice eh? :-)  Any comment?  Suggestion?  Insult?

Cheers,
Giuseppe

1) but the braves can find the current ugly hack here:
  http://it.gnu.org/~gscrivano/files/parallel_wget.patch