[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Wget-dev] wget2 | [Unverified] Wget2 _may_ be ignoring the robots f
|
From: |
Tim Rühsen |
|
Subject: |
Re: [Wget-dev] wget2 | [Unverified] Wget2 _may_ be ignoring the robots file on restart (#398) |
|
Date: |
Fri, 24 Aug 2018 10:47:57 +0000 |
Sounds like an easy test to add to our testsuite. Not unlikely that we don't
load + parse it from a local file.
> DO we even need to store the robots file ?
I would like to see it. It's part of a web site, so it might include relevant
information (e.g. sitemaps).
It shouldn't be saved when excluded by one of the accept/reject options.
--
Reply to this email directly or view it on GitLab:
https://gitlab.com/gnuwget/wget2/issues/398#note_96675923
You're receiving this email because of your account on gitlab.com.