Re: lynx-dev Re: clicktv search (was nothing)

[Mike Castle]

On Mon, Jun 28, 1999 at 02:45:58AM -0500, Mike Castle wrote:
> I did a tcpdump with NS and lynx and the results are:
> NS:
> POST http://www.clicktv.com/ctv30/results.cfm HTTP/1.0
> Referer: http://www.clicktv.com/ctv30/search.cfm?ctv_user_id=123456
> Proxy-Connection: Keep-Alive
> User-Agent: Mozilla/4.5 [en] (X11; I; Linux 2.0.36 i586; Nav)
> Host: www.clicktv.com
> Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
> Accept-Encoding: gzip
> Accept-Language: en
> Accept-Charset: iso-8859-1,*,utf-8
> Cookie: RCTVID=123456
> Content-type: application/x-www-form-urlencoded
> Content-length: 99
> 
> ctv_user_id=123456&date=06-27-99A07-11-99&time=&type_id=&keyword=xena&Chan_Ref=&SType_1=1&SType_2=1
> 
> To which clicktv sends back appropriate content...
> 
> However, for lynx:
> POST http://www.clicktv.com/results.cfm HTTP/1.0
> Host: www.clicktv.com
> Accept: text/html, text/plain, audio/x-realaudio, audio/x-pn-realaudio,
> text/sgml, video/mpeg, image/jpeg, image/tiff, image/x-rgb, image/png,
> image/x-xbitmap, image/x-xbm, image/gif, application/postscript, */*;q=0.01
> Accept-Encoding: gzip, compress
> Accept-Language: en
> Accept-Charset: en, iso-8859-1;q=0.01, us-ascii;q=0.01
> Pragma: no-cache
> Cache-Control: no-cache
> User-Agent: Lynx/2.8.3dev.2 libwww-FM/2.14
> Cookie2: $Version="1"
> Cookie: RCTVID=123456
> Content-type: application/x-www-form-urlencoded
> Content-length: 99
> 
> ctv_user_id=123456&date=06-28-99A07-12-99&time=&type_id=&keyword=xena&Chan_Ref=&SType_1=1&SType_2=1
> 
> To which clicktv responds:
> HTTP/1.0 302 Moved Temporarily
> Server: Netscape-Enterprise/3.5.1
> Date: Mon, 28 Jun 1999 07:28:59 GMT
> Vary: Accept-language
> Content-type: magnus-internal/cold-fusion
> Location: http://www.clicktv.com/search.cfm?ctv_user_id=123456
> 
> Note the lack of Referer: with Lynx.  The site may be broken and require
> referers (I faked a referer: with wget, but like I said, I had to use GET
> instead of POST).
> 
> Tomorrow evening I will install the appropriate perl modules, and try to
> fake the exact same headers with that, and see if that makes a difference.


Well, I installed the perl libwww-perl package and other necessary
support modules (HTML-Parser, MIME-Base64, and URI).  Fortunately, it comes
with some  nifty example applications, one of which is lwp-request.

I used the following as input (long line, wrapped at \'s):

ctv_user_id=123456&date=06-27-99A07-11-99&time=&type_id=&keyword=xena&\
Chan_Ref=&SType_1=1&SType_2=1

And the command line:

lwp-request -m POST http://www.clicktv.com/results.cfm < input > output.html

That failed as we usually see.

S:, I ran again as:

lwp-request \
-H 'Referer: http://www.clicktv.com/ctv30/search.cfm?ctv_user_id=123456' \
-m POST http://www.clicktv.com/results.cfm < input > output.html

And that, of course, worked.


The tcpdump output was:

POST http://www.clicktv.com/results.cfm HTTP/1.0
Host: www.clicktv.com
Referer: http://www.clicktv.com/ctv30/search.cfm?ctv_user_id=123456
User-Agent: lwp-request/1.38
Content-Length: 100
Content-Type: application/x-www-form-urlencoded

ctv_user_id=123456&date=06-27-99A07-11-99&time=&type_id=&keyword=xena&Chan_Ref=&SType_1=1&SType_2=1^J



So, obviously this site fails to work because of the lack of Referer:
header.

Btw, taking hint from one of Klaus' previous posts, I did the referer
as:

-H 'Referer: http://www.clicktv.com/ctv30/search.cfm' \

And that worked as well.

So, this could be more incentive to add multiple levels of Referer: for
GET and POST data (none, partial (stopping at ?), full).

Klaus, do you have any URL handy that discusses the security aspects of
this that could be passed on to the managers of that web site?

I'll be looking through my email archives to see if you posted anything
before, but if you had something handy....

mrc
-- 
       Mike Castle       Life is like a clock:  You can work constantly
  address@hidden  and be right all the time, or not work at all
www.netcom.com/~dalgoda/ and be right at least twice a day.  -- mrc
    We are all of us living in the shadow of Manhattan.  -- Watchmen