[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Wget-dev] wget2 | [Unverified] Wget2 _may_ be ignoring the robots f

From: Tim Rühsen
Subject: Re: [Wget-dev] wget2 | [Unverified] Wget2 _may_ be ignoring the robots file on restart (#398)
Date: Fri, 24 Aug 2018 10:47:57 +0000

Sounds like an easy test to add to our testsuite. Not unlikely that we don't 
load + parse it from a local file.

> DO we even need to store the robots file ?

I would like to see it. It's part of a web site, so it might include relevant 
information (e.g. sitemaps).
It shouldn't be saved when excluded by one of the accept/reject options.

Reply to this email directly or view it on GitLab: 
You're receiving this email because of your account on gitlab.com.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]