[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
wget2 | Check if robots.txt parser comlies with standardized format (#60
From: |
@rockdaboot |
Subject: |
wget2 | Check if robots.txt parser comlies with standardized format (#607) |
Date: |
Sat, 02 Jul 2022 11:05:47 +0000 |
Tim Rühsen created an issue: https://gitlab.com/gnuwget/wget2/-/issues/607
At https://github.com/rockdaboot/wget2/issues/263 we came up with the ida of
checking the existing parser against the robots.txt proposed standard..
Some people at Google created an RFC draft. I guess it won't change much until
it is finalized, so we can create a new parser.
See https://datatracker.ietf.org/doc/draft-koster-rep/
--
Reply to this email directly or view it on GitLab:
https://gitlab.com/gnuwget/wget2/-/issues/607
You're receiving this email because of your account on gitlab.com.
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- wget2 | Check if robots.txt parser comlies with standardized format (#607),
@rockdaboot <=