bug#62491: [berlin] certbot renewal appears to be broken

[Maxim Cournoyer]

Hi,

The TLS cert of https://disarchive.guix.gnu.org/ expired today.  Looking
at /var/log/mcron.log on Berlin, we see that the last certbot renew job
failed like so:

--8<---------------cut here---------------start------------->8---
2023-03-24 00:30:00 127768 certbot renew --webroot --webroot-path /var/www: 
running...
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
Saving debug log to /var/log/letsencrypt/letsencrypt.log
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/bootstrappable.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
Certificate not yet due for renewal
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/ci.guix.gnu.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
Certificate not yet due for renewal
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/disarchive.guix.gnu.org.conf
2023-03-24 00:30:02 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:32:54 127768 certbot renew --webroot --webroot-path /var/www: 
Renewing an existing certificate for disarchive.guix.gnu.org
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
Certbot failed to authenticate some domains (authenticator: webroot). The 
Certificate Authority reported these problems:
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   
Domain: disarchive.guix.gnu.org
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   
Type:   unauthorized
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www:   
Detail: 141.80.181.40: Invalid response from 
https://disarchive.guix.gnu.org/.well-known/acme-challenge/O1kK3tsJtH0r9RwvbCIFhHagJhBwewV3Ka0NPW86nAI:
 404
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
Hint: The Certificate Authority failed to download the temporary challenge 
files created by Certbot. Ensure that the listed domains serve their content 
from the provided --webroot-path/-w and that files created there can be 
downloaded from the internet.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
Failed to renew certificate disarchive.guix.gnu.org with error: Some challenges 
have failed.
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/dump.guix.gnu.org.conf
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
Certificate not yet due for renewal
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/guix.gnu.org.conf
2023-03-24 00:33:09 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:10 127768 certbot renew --webroot --webroot-path /var/www: 
Renewing an existing certificate for guix.gnu.org
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
Certbot failed to authenticate some domains (authenticator: webroot). The 
Certificate Authority reported these problems:
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   
Domain: guix.gnu.org
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   
Type:   unauthorized
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www:   
Detail: 2a0c:e300::58: Invalid response from 
https://guix.gnu.org/.well-known/acme-challenge/_PlXq5i2BRw23Ui1Yl4rLtyB2aSDnUNMZXurCWBwH-k:
 404
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
Hint: The Certificate Authority failed to download the temporary challenge 
files created by Certbot. Ensure that the listed domains serve their content 
from the provided --webroot-path/-w and that files created there can be 
downloaded from the internet.
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
Failed to renew certificate guix.gnu.org with error: Some challenges have 
failed.
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/guix.info.conf
2023-03-24 00:33:18 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:19 127768 certbot renew --webroot --webroot-path /var/www: 
Renewing an existing certificate for guix.info and www.guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
Certbot failed to authenticate some domains (authenticator: webroot). The 
Certificate Authority reported these problems:
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   
Domain: guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   
Type:   unauthorized
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   
Detail: 141.80.181.40: Invalid response from 
https://guix.gnu.org/.well-known/acme-challenge/O6y6aqSvLdjdS77MgaEhh7sN7Q75OQX3Jz69xnT4qnY:
 404
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   
Domain: www.guix.info
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   
Type:   unauthorized
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www:   
Detail: 141.80.181.40: Invalid response from 
https://guix.gnu.org/.well-known/acme-challenge/lCioloihdJF6xwwTBg6cSNFjRearp4EBZBWcjkznrUE:
 404
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
Hint: The Certificate Authority failed to download the temporary challenge 
files created by Certbot. Ensure that the listed domains serve their content 
from the provided --webroot-path/-w and that files created there can be 
downloaded from the internet.
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
Failed to renew certificate guix.info with error: Some challenges have failed.
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/issues.guix.gnu.org.conf
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
Certificate not yet due for renewal
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/issues.guix.info.conf
2023-03-24 00:33:25 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:26 127768 certbot renew --webroot --webroot-path /var/www: 
Renewing an existing certificate for issues.guix.info and 3 more domains
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
Certbot failed to authenticate some domains (authenticator: webroot). The 
Certificate Authority reported these problems:
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   
Domain: guix.info
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   
Type:   unauthorized
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www:   
Detail: 141.80.181.40: Invalid response from 
https://guix.gnu.org/.well-known/acme-challenge/Yv4KpoYC95LzGsM5IPTE68vf6lLfNHVK5kMUocSuDW0:
 404
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
Hint: The Certificate Authority failed to download the temporary challenge 
files created by Certbot. Ensure that the listed domains serve their content 
from the provided --webroot-path/-w and that files created there can be 
downloaded from the internet.
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
Failed to renew certificate issues.guix.info with error: Some challenges have 
failed.
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/monitor.guix.gnu.org.conf
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:39 127768 certbot renew --webroot --webroot-path /var/www: 
Renewing an existing certificate for monitor.guix.gnu.org
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
Certbot failed to authenticate some domains (authenticator: webroot). The 
Certificate Authority reported these problems:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
Domain: monitor.guix.gnu.org
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
Type:   unauthorized
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
Detail: 141.80.181.40: Invalid response from 
https://monitor.guix.gnu.org/.well-known/acme-challenge/_wxH92e9QQag7TEYdqsA4-C-5pE5DnUd6pzMvQWzWNU:
 400
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
Hint: The Certificate Authority failed to download the temporary challenge 
files created by Certbot. Ensure that the listed domains serve their content 
from the provided --webroot-path/-w and that files created there can be 
downloaded from the internet.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
Failed to renew certificate monitor.guix.gnu.org with error: Some challenges 
have failed.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/www.guixwl.org-0001.conf
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
Certificate not yet due for renewal
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
Processing /etc/letsencrypt/renewal/www.guixwl.org.conf
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
Certificate not yet due for renewal
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: The 
following certificates are not due for renewal yet:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/bootstrappable.org/fullchain.pem expires on 2023-05-14 
(skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/ci.guix.gnu.org/fullchain.pem expires on 2023-06-04 
(skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/dump.guix.gnu.org/fullchain.pem expires on 2023-06-04 
(skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/issues.guix.gnu.org/fullchain.pem expires on 2023-06-04 
(skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/www.guixwl.org-0001/fullchain.pem expires on 2023-06-04 
(skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/www.guixwl.org/fullchain.pem expires on 2023-06-04 
(skipped)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: All 
renewals failed. The following certificates could not be renewed:
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/disarchive.guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/guix.info/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/issues.guix.info/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www:   
/etc/letsencrypt/live/monitor.guix.gnu.org/fullchain.pem (failure)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: - - 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 5 
renew failure(s), 0 parse failure(s)
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: Ask 
for help or search for solutions at https://community.letsencrypt.org. See the 
logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more 
details.
2023-03-24 00:33:54 127768 certbot renew --webroot --webroot-path /var/www: 
failed after 234.635s with: (misc-error #f unclean exit status ~S (1) 
#f)--8<---------------cut here---------------end--------------->8---

I removed the certbot file name prefix
(/gnu/store/jnp0166xw62dafd2zgxdmvjb6yq8ak32-certbot-1.28.0/bin/) in the
above output to improve readability.

-- 
Thanks,
Maxim