[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Sks-devel] Request: Install an efficient robots.txt file
From: |
Kristian Fiskerstrand |
Subject: |
Re: [Sks-devel] Request: Install an efficient robots.txt file |
Date: |
Thu, 22 Jun 2017 13:08:06 +0200 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.1.1 |
On 06/22/2017 10:40 AM, robots.txt fan wrote:
> Kristian, you have responded to this thread, I believe you manage the first
> one on the list. Is there a reason why only /status is blocked and not /pks?
>
> https://sks-keyservers.net (blocks /status, but not /pks)
The real reason is that /pks didn't exist when the robots.txt file was
created, so I've [added it now], granted more for site resource
management reasons than privacy reasons.
From a privacy perspective robots.txt doesn't make sense, the data is
already public, bad actors ignore robots.txt and crawl the site just the
same; and the full data set is available and part of regular workflow
for bootstrapping own servers.
References:
[added it now]
https://git.sumptuouscapital.com/?p=sks-keyservers-pool.git;a=commit;h=b98e7522990961541165dfc23781a45a1a5e05a9
--
----------------------------
Kristian Fiskerstrand
Blog: https://blog.sumptuouscapital.com
Twitter: @krifisk
----------------------------
Public OpenPGP keyblock at hkp://pool.sks-keyservers.net
fpr:94CB AFDD 3034 5109 5618 35AA 0B7F 8B60 E3ED FAE3
----------------------------
"Whatever you do in life, surround yourself with smart people who'll
argue with you."
John Wooden
signature.asc
Description: OpenPGP digital signature
- Re: [Sks-devel] Request: Install an efficient robots.txt file, (continued)
- Re: [Sks-devel] Request: Install an efficient robots.txt file, robots.txt fan, 2017/06/20
- Re: [Sks-devel] Request: Install an efficient robots.txt file, Gabor Kiss, 2017/06/20
- Re: [Sks-devel] Request: Install an efficient robots.txt file, Robert J. Hansen, 2017/06/20
- Re: [Sks-devel] Request: Install an efficient robots.txt file, Ari Trachtenberg, 2017/06/20
- Re: [Sks-devel] Request: Install an efficient robots.txt file, Kristian Fiskerstrand, 2017/06/20
- Re: [Sks-devel] Request: Install an efficient robots.txt file, Ari Trachtenberg, 2017/06/20
- Re: [Sks-devel] Request: Install an efficient robots.txt file, Kristian Fiskerstrand, 2017/06/20
Re: [Sks-devel] Request: Install an efficient robots.txt file, Paul M Furley, 2017/06/20
Re: [Sks-devel] Request: Install an efficient robots.txt file, m, 2017/06/20
Re: [Sks-devel] Request: Install an efficient robots.txt file, Dennis Fink, 2017/06/22
Re: [Sks-devel] Request: Install an efficient robots.txt file, Daniel Austin, 2017/06/24
Message not availableRe: [Sks-devel] Request: Install an efficient robots.txt file, robots.txt fan, 2017/06/25