|
| From: | Leo Famulari |
| Subject: | bug#52338: Crawler bots are downloading substitutes |
| Date: | Mon, 6 Dec 2021 16:20:55 -0500 |
I noticed that some bots are downloading substitutes from ci.guix.gnu.org. We should add a robots.txt file to reduce this waste. Specifically, I see bots from Bing and Semrush: https://www.bing.com/bingbot.htm https://www.semrush.com/bot.html
| [Prev in Thread] | Current Thread | [Next in Thread] |