[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: LYNX-DEV lynx-dev list status/problems
From: |
David Woolley |
Subject: |
Re: LYNX-DEV lynx-dev list status/problems |
Date: |
Mon, 14 Jul 1997 19:17:58 +0100 (BST) |
>
> Any sweep that comes across my FAQ gets into the message pages of
> the archive, and could crawl from there to the indices and all
> messages. Putting a form at the front door doesn't mean that
> there aren't back doors.
Except that I think the archives should be visible to the crawlers from
the search engines, you should be using robots.txt to deter them, and
if you have enough control, you could try an infinitely recursive page,
generated by CGI, to inconvenience anyone who ignores robots.txt.
;
; To UNSUBSCRIBE: Send a mail message to address@hidden
; with "unsubscribe lynx-dev" (without the
; quotation marks) on a line by itself.
;
- Re: LYNX-DEV lynx-dev list status/problems, (continued)
Re: LYNX-DEV lynx-dev list status/problems, tzeruch, 1997/07/12
Re: LYNX-DEV lynx-dev list status/problems, Jay W. Meeuwig, 1997/07/12
Re: LYNX-DEV lynx-dev list status/problems, Laura Eaves, 1997/07/12
Re: LYNX-DEV lynx-dev list status/problems, Jim Dennis, 1997/07/12
Re: LYNX-DEV lynx-dev list status/problems, Philip Webb, 1997/07/14
Re: LYNX-DEV lynx-dev list status/problems, Al Gilman, 1997/07/14
Re: LYNX-DEV lynx-dev list status/problems, Klaus Weide, 1997/07/14
Re: LYNX-DEV lynx-dev list status/problems, Jonathan Sergent, 1997/07/14
Re: LYNX-DEV lynx-dev list status/problems, Al Gilman, 1997/07/14
LYNX-DEV filter policies, Al Gilman, 1997/07/14
Re: LYNX-DEV lynx-dev list status/problems, Jim Dennis, 1997/07/14
Re: LYNX-DEV lynx-dev list status/problems, Jim Dennis, 1997/07/14
Re: LYNX-DEV lynx-dev list status/problems, Nelson Henry Eric, 1997/07/14