Re: LYNX-DEV Downloading a whole web site for local offline browsing

[Filip M Gieszczykiewicz]

You (Peter D Read) wrote:
> I have been searching for some while for something that will
> allow me to download a complete web site, so that I can browse
> the contents locally at will.
> There is a commercial application "Black Widow" that does this
> for Windows but I am Unix based.
[zap]

This is going to set some of you "off" but I've been getting hit hard
by the bozos who download our whole site (~20MB of text)... we get
about 19 of those a week (on average - ie. 19 complete "sucks" from
the title page down to the last stupid icon used on a well-hidden
page that no one in their right mind has ever dowloaded before).

I'm _seriously_ thinking about "interlocking" portions of my site
with CGI's that ask you to check off some random box that you
want to proceed to next "tree" of documents. I don't have a problem
with people who *consciously* download the whole site (though we
have mirrors for "those" excuses and offere "off-line" ZIPped
versions too) but I *DO* have a *BIG* problem with people who
do it with automated and braindead tools just to save themselves
some time... and guess who pays for that? Yup... the provider.

That's all right if it's a commercial site... it's NOT ok when
your bill goes up if your traffic goes significantly up. We've
gone from 65K hits/week (800MB out) to 130K hits/week with 1.6GB
out... I can't _wait_ for my next increase... Joy.

Not a _direct_ flame to people asking for this... mind you... just
please _consider_ that if you write any tools that do this. Things
like MOMspider at least _wait_ 5-10 seconds between fetches! Those
assholes from "Black Widow" and MicroShaft do NOT... so they
load down the server like there was no tomorrow. But, why not...
it's all free. May they roast in hell.

Take care.