Letsencrypt on Debian 10 – switching to new ‘snap’ based Certbot

Official instructions: https://certbot.eff.org/lets-encrypt/debianbuster-nginx

If you’ve upgraded a machine Debian 10 (Buster) or later you’ll see a warning like this:

/etc/cron.weekly/certbot:
Your system is not supported by certbot-auto anymore.
certbot-auto and its Certbot installation will no longer receive updates.
You will not receive any bug fixes including those fixing server compatibility
or security problems.
Please visit https://certbot.eff.org/ to check for other alternatives.

Certbot, the tool for updating Let’s Encrypt certificates, is now distributed via Snap.  If you haven’t heard of that, Snap is a new package format – the point is it’s more cross platform – so a single package will currently support Ubuntu, Debian, Fedora, Arch Linux, Manjaro, and CentOS/RHEL.  More details >

This means you need to:

then, typically you already have a weekly cron file that calls certbot to renew the certificates, and restarts your apache or nginx server (the latter step is important because otherwise it’ll keep serving a cached copy of the old certificate, which will eventually expire).

Here’s the (old) contents of my /etc/cron.weekly/certbot:

#!/bin/bash
/home/william/letsencrypt/certbot-auto renew
service nginx reload

You need to remove the line calling certbot-auto, but leave the one restarting nginx once a week.  You might choose to rename the cron file or add a comment to make it clearer to your future self why it’s needed.

Next, check snap is to up to date:

sudo snap install core; sudo snap refresh core

Install the certbot package:

sudo snap install --classic certbot

Then you can test a manual update either:

sudo certbot certonly --nginx

(on the interactive version you’re presented with a list of sites to choose from)

Finally, do a renewal “dry-run” to check for problems:

sudo certbot renew --dry-run

The key points about dry-runs:

  • they use the Let’s Encrypt staging environment, so you don’t hit rate limits
  • therefore, real certificates aren’t requested or saved
  • it’s basically testing that Let’s Encrypt end is able successfully handshake with your server, can write files to correct places etc.

Renewals

Renewals can use one of three methods: crontab, cron files or systemctl timers.

On the servers I’ve installed so far, certbot has automatically setup a timer. You can verify it has by running systemctl as follows:

$ systemctl list-timers
NEXT LEFT LAST PASSED UNIT ACTIVATES
Mon 2021-02-15 12:12:00 GMT 1h 43min left Mon 2021-02-15 07:57:03 GMT 2h 31min ago snap.certbot.renew.timer snap.certbot.renew.service
```

So certificate refreshes will happen automatically, but remember you need to handle regularly restarting your web server (and weekly is as good an interval as any to do that).

Here’s another link to the documentation:
https://certbot.eff.org/lets-encrypt/debianbuster-nginx

Fixing invalid public key for packages.sury.org

Update 21 March 2021: there’s been a recent uptick in traffic to this page – the current problem everyone is having is due to an expired key.  Read more in this deb.sury.org github issue.

Solution (Debian 10.8 Buster):

(comments in various places suggest removing the old key is crucial)

sudo apt-key del B188E2B695BD4743
sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg

Useful command worth noting:

apt-key list

Original April 2019 blog post:

If you’re running Debian and using:

deb https://packages.sury.org/php/ stretch main

(it might be in /etc/apt/sources.list.d/php.list rather than the usual sources.list)

… you may see this error:

Err:5 https://packages.sury.org/php stretch InRelease
The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B188E2B695BD4743
Reading package lists... Done
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.sury.org/php stretch InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B188E2B695BD4743
W: Failed to fetch https://packages.sury.org/php/dists/stretch/InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B188E2B695BD4743
W: Some index files failed to download. They have been ignored, or old ones used instead.

This isn’t widely blogged yet, however the best source of info is the Issue queue for the deb.sury.org GitHub repository – it turns out that in mid-March, the key for each repository on sury.org was regenerated due to a compromised server.

Here’s the command to download the new one, after which apt will work as expected.

sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg

 

HowTo: Nginx with HTTP2 support on Debian Jessie

The original problem:

  • SPDY has been replaced by HTTP2, which is better in a number of ways
  • As of June 2016, Chrome has dropped support for SPDY
  • HTTP2 uses ALPN
  • ALPN requires OpenSSL 1.0.2
  • Debian Stable (aka Jessie aka v8) and others OSes only had 1.0.1

What’s changed:

Procedure:

  • Add jessie-backports and jessie-nginx-http2 (ansible playbook)
  • Upgrade openssl them from the correct place: sudo apt-get install -t jessie-backports openssl
  • sudo apt-get install nginx-full (which should pull in various libnginx-mod packages)
  • Change any references in your Nginx config files from spdy to http2
  • Run sudo nginx -t to verify the configuration is valid
  • Start server

Verify HTTP2 is working (Chrome or Opera):

Developer Tools, Network tab, reload page, enable the Protocol column, look for H2, which means HTTP2.

Extra step for LetsEncrypt / Certbot compatibility:

A few days after doing this I got the following error when my weekly cronjob for renewing LetsEncrypt certificates ran:

    build/temp.linux-x86_64-2.7/_openssl.c:415:30: fatal error: openssl/opensslv.h: No such file or directory
     #include <openssl/opensslv.h>

And on running it manually I had this:

The following packages have unmet dependencies:
 libssl-dev : Depends: libssl1.0.0 (= 1.0.1t-1+deb8u6) but 1.0.2k-1~bpo8+1 is to be installed
 Recommends: libssl-doc but it is not going to be installed

The solution was just to pull in libssl-dev from jessie-backports too:

apt-get install -t jessie-backports libssl-dev

Note, in my case, I have a git clone of certbot rather than a packaged version, though it is now available as a backport for Debian Jessie.

backup2l troubleshooting – skipcond not ignoring paths

Something to watch out for…

backup2l, the popular Unix backup software, has a setting in /etc/backup2l.conf called skipcond that lets you ignore files/paths.  It uses find syntax.

You might have written something like this if you don’t want to both backing up old logs, say:

SKIPCOND=(-path /var/www/mysite/system/cms/logs/log*" -o -name "*.o")

But when you run backup2l -e to simulate the backup, it says:

856 / 43210 file(s), 13 / 5003 dir(s), 2.2GB / 4.3GB (uncompressed)
skipping: 0 file(s), 0 dir(s), 0 B (uncompressed)

You know this is wrong for two reasons, the overall size is too big and it’s not skipping anything.

This could be because of symlinks.  To debug, run a real backup (with the -b switch), and use backup2l -l [pattern] (where pattern is some file(s) you know are in the directory you want to exclude) to see what’s there.

You may find it’s actually backing up /usr/share/nginx/mysite, because /var/www is symlinked to it:

lrwxrwxrwx  1 root root    17 Aug  2  2015 www -> /usr/share/nginx/

Remember backup2l has a purge (-p) option that lets you remove individual differential or incremental backups by specifying the number. So if you’ve just run all.1375, say, you can delete that with -p 1375 and when you run it again it’ll reuse the number.

 

Chrome, SPDY, HTTP/2, Nginx, NPN, APLN, OpenSSL and Debian

Mattias Geniar has written this up in some detail, but to summarise:

Nginx 1.10 (the new major, stable version) has replaced SPDY with HTTP/2. But as of this week, Chrome now only supports HTTP/2 using ALPN.  ALPN requires OpenSSL 1.0.2.  But stable Debian (and CentOS, and other flavours of Linux) only have 1.0.1 and, right now (it seems to me)  there’s not a lot of hope Debian will back-port the new version: many other services also use OpenSSL, they’d all need to be checked/updated…

If you’re using DotDeb on Jessie (aka Debian 8), the nginx packages (nginx nginx-common nginx-full etc.) will be held back when running apt-get upgrade.

What should I do?  Updated – SOLUTION AVAILABLE! (2 March 2017) A newer OpenSSL has been backported.  You need to add some extra repositories and use a special command to install it, but that’s it. Full Instructions

Fixing the MailChimp API SSL certificate

Here’s a StackOverflow answer by me for anyone who sees this error:

SSL certificate problem: unable to get local issuer certificate

…when using the MailChimp API.

The problem is due to MC using a root certificate that’s been removed from the Mozilla certificate bundle (you’re most likely to see it on Debian systems.)

This is a safer way to work around the problem that rolling all your certificates back to 2014 or disabling SSL certificate verification entirely.  You also won’t need to set cURL options or edit your php.ini.

Also covered: how to ‘ping’ the MailChimp API to check it’s working.

Patching servers for glibc (CVE-2015-7547) – remember to reboot!

This is a significant (though we’re not sure yet realistically how dangerous) buffer overflow bug in the Linux C library, glibc. The library handles system calls, for example, reading/writing to files.

The bug is with gethostbyname (used to find the DNS entry for a domain name).  Full explanation (Google)

Security updates have been available since 16 Feb 2016.

It’s easy to update but… remember to reboot afterwards! That’s the main reason I wrote this up – there’s a note to this effect in the security advisories, but utilities like aptitude won’t remind you. All sorts of services could be using glibc and remain vulnerable until they’ve restarted. Dan Kaminsky, at length, on how bad it could be

Much as it’s nice to accumulate a large uptime value, a forced reboot is also a useful opportunity to check the relevant services on your machine will come up after a power outage and, for example, your firewall settings are being loaded correctly.

Recommendation: check your existing version before you update, so you can verify it’s changed afterwards:

sudo aptitude show libc6 (Debian)
or execute the library itself, e.g.
/lib64/libc.so.6
(look for the line: Compiled on a Linux 2.6.32 system on [date])

How To:

Use your usual update procedure (e.g. apt-get update/upgrade on Debian/Ubuntu, yum update on CentOS/RedHat) then reboot the server.

Remember to do your VMs (e.g. Vagrant boxes) too.

Note packages are backported, so for Debian you pay attention to the uxx after the version number, not the version itself.

Does it affect Mac OS X?

No. (OS X doesn’t have glibc.)