wget-dev
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

wget2 | Check if robots.txt parser comlies with standardized format (#60


From: @rockdaboot
Subject: wget2 | Check if robots.txt parser comlies with standardized format (#607)
Date: Sat, 02 Jul 2022 11:05:47 +0000


Tim Rühsen created an issue: https://gitlab.com/gnuwget/wget2/-/issues/607



At https://github.com/rockdaboot/wget2/issues/263 we came up with the ida of 
checking the existing parser against the robots.txt proposed standard..

Some people at Google created an RFC draft. I guess it won't change much until 
it is finalized, so we can create a new parser.

See https://datatracker.ietf.org/doc/draft-koster-rep/

-- 
Reply to this email directly or view it on GitLab: 
https://gitlab.com/gnuwget/wget2/-/issues/607
You're receiving this email because of your account on gitlab.com.




reply via email to

[Prev in Thread] Current Thread [Next in Thread]