HowTo: Nginx with HTTP2 support on Debian Jessie

The original problem:

  • SPDY has been replaced by HTTP2, which is better in a number of ways
  • As of June 2016, Chrome has dropped support for SPDY
  • HTTP2 uses ALPN
  • ALPN requires OpenSSL 1.0.2
  • Debian Stable (aka Jessie aka v8) and others OSes only had 1.0.1

What’s changed:

Procedure:

  • Add jessie-backports and jessie-nginx-http2 (ansible playbook)
  • Upgrade openssl them from the correct place: sudo apt-get install -t jessie-backports openssl
  • sudo apt-get install nginx-full (which should pull in various libnginx-mod packages)
  • Change any references in your Nginx config files from spdy to http2
  • Run sudo nginx -t to verify the configuration is valid
  • Start server

Verify HTTP2 is working (Chrome or Opera):

Developer Tools, Network tab, reload page, enable the Protocol column, look for H2, which means HTTP2.

Extra step for LetsEncrypt / Certbot compatibility:

A few days after doing this I got the following error when my weekly cronjob for renewing LetsEncrypt certificates ran:

    build/temp.linux-x86_64-2.7/_openssl.c:415:30: fatal error: openssl/opensslv.h: No such file or directory
     #include <openssl/opensslv.h>

And on running it manually I had this:

The following packages have unmet dependencies:
 libssl-dev : Depends: libssl1.0.0 (= 1.0.1t-1+deb8u6) but 1.0.2k-1~bpo8+1 is to be installed
 Recommends: libssl-doc but it is not going to be installed

The solution was just to pull in libssl-dev from jessie-backports too:

apt-get install -t jessie-backports libssl-dev

Note, in my case, I have a git clone of certbot rather than a packaged version, though it is now available as a backport for Debian Jessie.

backup2l troubleshooting – skipcond not ignoring paths

Something to watch out for…

backup2l, the popular Unix backup software, has a setting in /etc/backup2l.conf called skipcond that lets you ignore files/paths.  It uses find syntax.

You might have written something like this if you don’t want to both backing up old logs, say:

SKIPCOND=(-path /var/www/mysite/system/cms/logs/log*" -o -name "*.o")

But when you run backup2l -e to simulate the backup, it says:

856 / 43210 file(s), 13 / 5003 dir(s), 2.2GB / 4.3GB (uncompressed)
skipping: 0 file(s), 0 dir(s), 0 B (uncompressed)

You know this is wrong for two reasons, the overall size is too big and it’s not skipping anything.

This could be because of symlinks.  To debug, run a real backup (with the -b switch), and use backup2l -l [pattern] (where pattern is some file(s) you know are in the directory you want to exclude) to see what’s there.

You may find it’s actually backing up /usr/share/nginx/mysite, because /var/www is symlinked to it:

lrwxrwxrwx  1 root root    17 Aug  2  2015 www -> /usr/share/nginx/

Remember backup2l has a purge (-p) option that lets you remove individual differential or incremental backups by specifying the number. So if you’ve just run all.1375, say, you can delete that with -p 1375 and when you run it again it’ll reuse the number.

 

Chrome, SPDY, HTTP/2, Nginx, NPN, APLN, OpenSSL and Debian

Mattias Geniar has written this up in some detail, but to summarise:

Nginx 1.10 (the new major, stable version) has replaced SPDY with HTTP/2. But as of this week, Chrome now only supports HTTP/2 using ALPN.  ALPN requires OpenSSL 1.0.2.  But stable Debian (and CentOS, and other flavours of Linux) only have 1.0.1 and, right now (it seems to me)  there’s not a lot of hope Debian will back-port the new version: many other services also use OpenSSL, they’d all need to be checked/updated…

If you’re using DotDeb on Jessie (aka Debian 8), the nginx packages (nginx nginx-common nginx-full etc.) will be held back when running apt-get upgrade.

What should I do?  Updated – SOLUTION AVAILABLE! (2 March 2017) A newer OpenSSL has been backported.  You need to add some extra repositories and use a special command to install it, but that’s it. Full Instructions

Fixing the MailChimp API SSL certificate

Here’s a StackOverflow answer by me for anyone who sees this error:

SSL certificate problem: unable to get local issuer certificate

…when using the MailChimp API.

The problem is due to MC using a root certificate that’s been removed from the Mozilla certificate bundle (you’re most likely to see it on Debian systems.)

This is a safer way to work around the problem that rolling all your certificates back to 2014 or disabling SSL certificate verification entirely.  You also won’t need to set cURL options or edit your php.ini.

Also covered: how to ‘ping’ the MailChimp API to check it’s working.

Patching servers for glibc (CVE-2015-7547) – remember to reboot!

This is a significant (though we’re not sure yet realistically how dangerous) buffer overflow bug in the Linux C library, glibc. The library handles system calls, for example, reading/writing to files.

The bug is with gethostbyname (used to find the DNS entry for a domain name).  Full explanation (Google)

Security updates have been available since 16 Feb 2016.

It’s easy to update but… remember to reboot afterwards! That’s the main reason I wrote this up – there’s a note to this effect in the security advisories, but utilities like aptitude won’t remind you. All sorts of services could be using glibc and remain vulnerable until they’ve restarted. Dan Kaminsky, at length, on how bad it could be

Much as it’s nice to accumulate a large uptime value, a forced reboot is also a useful opportunity to check the relevant services on your machine will come up after a power outage and, for example, your firewall settings are being loaded correctly.

Recommendation: check your existing version before you update, so you can verify it’s changed afterwards:

sudo aptitude show libc6 (Debian)
or execute the library itself, e.g.
/lib64/libc.so.6
(look for the line: Compiled on a Linux 2.6.32 system on [date])

How To:

Use your usual update procedure (e.g. apt-get update/upgrade on Debian/Ubuntu, yum update on CentOS/RedHat) then reboot the server.

Remember to do your VMs (e.g. Vagrant boxes) too.

Note packages are backported, so for Debian you pay attention to the uxx after the version number, not the version itself.

Does it affect Mac OS X?

No. (OS X doesn’t have glibc.)