WordPress REST API security risks

I’d recommend any WordPress users install the disable-JSON-API plugin.  This will prevent anonymous access to WP REST API endpoints, such as /wp-json/wp/v2/users – which provides a list of usernames (API docs).

The REST API was introduced in WordPress 4.7 and is (unfortunately) on by default with no option in settings to turn it off – the idea is it will in time be used for AJAX in some of the admin system, not just external requests.   There is a good change if you were running 4.7.0 or 4.7.1 and did not immediately upgrade to the 4.7.2 security release posts may have been defaced.

Also be aware that the WordPress readme.html file no longer displays complete version numbers – e.g. it will show 4.7 rather than 4.7.2.  (a step backward, IMHO).  Though you might not be aware however you can get the version number by clicking on the WordPress icon at the left of the admin bar (and of course, WP-CLI users can do: wp core version)

Tips: Updating the Adobe Flash plugin for Firefox, Safari, Opera on Mac

I’ve routinely found the standard Adobe Flash player updater for Mac (OS X/macOS) fails right at the end, without explanation.

However there is a version that always works. Go to:

https://helpx.adobe.com/flash-player/kb/installation-problems-flash-player-mac.html

… and choose Flash player for Safari and Firefox – NPAPI – the download URL is normally:

https://fpdownload.macromedia.com/pub/flashplayer/latest/help/install_flash_player_osx.dmg

The opening dialog box should look a bit different – like this:

Adobe Flash Installer

Opera users

Use this version (last tested 10 Feb 2017):

https://fpdownload.macromedia.com/pub/flashplayer/latest/help/install_flash_player_osx_ppapi.dmg

Firefox plugin checker

This is a very fast and handy way of seeing if your copy of Flash or Silverlight is up to date.

https://www.mozilla.org/en-US/plugincheck/

Let’s Encrypt news – client name change and auto renewals

Updated Wed 14 Sep 2016 (new installation guide URL, clarify name change.)

Let’s Encrypt have updated their getting started page, but the following may help anyone trying to understand the latest changes.

If you’re installing from scratch, use Certbot (see below) and start here.  You’ll get custom instructions for your operating system and web server – the client can now be installed via a package on newer systems.

The name

The client (now at version 0.6.0) from letsencrypt-auto to Certbot – to be precise, the project/ecosystem is still called Let’s Encrypt, while Certbot is the EFF’s certificate deployment client.

You’ll find that certbot-auto (a shell script) is an exact copy of the letsencrypt-auto, so all previous commands will still work.

The git repository has also been renamed – the old one is redirected.

Old: https://github.com/letsencrypt/letsencrypt
New: https://github.com/certbot/certbot

You can update the location of your ‘origin’ remote in .git/config

Renewals

You can now renew all your certificates at once.

# to test
~/letsencrypt/certbot-auto renew --dry-run

# to actually do it
~/letsencrypt/certbot-auto renew

There’s some clever stuff going on here:

  • It uses all your previous settings.
  • It renews any certificates that will expire within 30 days.
  • Afterwards you get a list of which were renewed and which were skipped (“not due for renewal yet”)
  • --dry-run  use staging server, so doesn’t count towards API limits.
  • “renew” is designed for unattended use.
  • Remember you still need to reload apache/nginx afterwards.

Example of a cron.weekly script you could use:

#!/bin/bash
/path/to/letsencrypt/certbot-auto renew
service nginx reload

Tip: Firefox address bar, avoiding phishing scams

Paul Moore tweeted this demonstration of a rather devious phishing scam earlier, where someone has taken advantage of the fact a capital I can look virtually identical to a lower-case L in sans-serif fonts* (and therefore a hacker can register a domain with the incorrect spelling and provide a link that tricks people.). Letters can also be paired: an adjacent r and n make a convincing m if you’re not looking closely.

There is some protection in browsers against this nowadays: as soon as you press return (though not as you type or paste the URL in) the entire address is converted to lower case. In Firefox, the domain is highlighted in black while the rest – including, crucially, any subdomains that precede it – is greyed out, providing a further hint to the site’s real identity.

It’s common for phishing scams to use multiple subdomains to make you think you’re on the genuine site – e.g https://www.mytrustworthybank.com.somewhere.suspicio.us/login

You can go a step further.  The Firefox address bar can be customised via the userChrome.css CSS file –on a Mac you need to go to ~/Library/Application Support/Firefox/Profiles/[your profile]/chrome and create a copy based on userChrome-example.css

Add the following and restart:

#urlbar {
    font-size: 21px !important;
    font-family: serif !important;
}

This will make the font bigger and use the default serif font rather than a sans-serif one.  I and L are easy to distinguish and, for web developers, having a larger typeface probably helps, regardless of how good your eyesight is – we look at URLs more often than most people, after all. It ought to reduce the number of typos you make.

Firefox address bar

There are Add-ons that do similar things if you don’t want to write your own CSS.

GDS, the Government Digital Service, use a specially designed sans-serif font called New Transport based on Kinneir and Calvert’s work on British road signs.  Note the legibility clues, especially the curve at the bottom of the lower-case L.

Over time we’ve recognised text inputs are better when they’re bigger and made it easier for users to change the zoom level, but the address bar has remained resolutely small and in a font that’s not as legible as it might be. Perhaps we should fix that.

* It can happen the other way round too (in this case it was Sky’s fault for forcing a screenshot into the wrong aspect ratio, making a lower-case L slightly fatter and easily mistaken for a lower-case I, particular after it’s been bounced sent through the vision mixer and the presenter has to read it off a screen several metres away.)

OS X El Capitan upgrade tips

(updated Sun 3 April 2016)

Yesterday I upgraded my primary computer (a 2012 Mac Mini) to El Capitan (OS X 10.11).

It’s normal for me to wait 6 months or so for the .3 or .4 OS X release, to allow Apple to fix hardware, networking, performance problems or random bugs and – crucially – for other developers to do the same with their applications (by no means everyone is actively testing software on the beta versions.) I’d recommend this to others.

Installation notes/tips:

  • Backup first (obviously.)
  • I recovered around 40GB of free space after installation (and 17GB on a Macbook Air upgraded soon after.)
  • performance is generally snappier (the CPU graph in Activity Monitor looks flatter when the system is idle, also considerable improvements when previewing files – not just PDFs but video as well)
  • performance will degrade considerably immediately after installation (less so on an SSD, but the Mac Mini’s HDD + Fusion Drive suffered a lot) as Spotlight reindexes everything (you’ll see sustained high disk IO and high CPU from md5 and associated processes.) If there is more than one user of the computer, this will happen the first time each user logs in, as each has a separate Spotlight database.  If you use Dropbox, temporarily quitting that will help it complete faster.  Keep Activity Monitor open and once indexing has finished, disk IO will return to zero.
  • I recommend a clean restart after that to check everything is ok.
  • You’ll need to upgrade the usual things, e.g. XCode, any Text to Speech voices you have installed.
  • Homebrew requires a change of ownership for /usr/local/ – see discussion on Stack Exchange – to the best of my knowledge chown -R is perfectly safe, but you certainly shouldn’t start messing around disabling SIP.
  • SuperDuper – a program that does disk backups and cloning – requires you delete and recreate any existing scheduled backups, otherwise they won’t run.  More info
  • Expect to do one large Time Machine backup afterwards (again, this was smaller on the Macbook Air.)

Software compatibility:

  • I only had one program that was incompatible, a version of GPG (encryption).
  • If you still have Photoshop CS4, it needs the old version of Java. This is painless – on attempting to run it a Dialog Box informs you of this, the More Info button links to an Apple support page with a direct download to the file. You just install it and it works straight away.
  • No issues at all with PhpStorm (Jetbrains had display problems last year because of java bugs.)

Security:

Previously, Apple developed two-step authentication, with El Capitan they added two-factor authentication.  The former is still supported, the latter is more secure – “It uses different methods to trust devices and deliver verification codes” – but it requires first turning 2-step off, adding security questions (note your answers are max 32 characters) and then setting up 2-factor on an iOS device (which’ll discard the security questions you just created.)  Instructions (9to5mac)

Note that, given the current Apple/US government iPhone case, if you can’t get in with two-factor there is a recovery process (unlike if you lose your FileVault recovery key, say) but it’s not immediate.  The KB article refers to a confirmation email to your registered account, possibly being required to confirm credit card details etc.

How to turn off Fail2Ban email notifications

Updated 23 Mar 2016 with corrections.
(These instructions based on a CentOS machine I’m responsible for.)

You may find yourself getting multiple emails per day from a server running Fail2Ban, each and every time it blocks an IP address after several failed SSH logins, e.g.

Subject: [Fail2Ban] SSH: banned 123.123.123.123 from myserver

It’s not terribly obvious how to disable these – you’ll find plenty of threads from people asking how to turn Fail2Ban notifications on, not so many asking how to turn them off, also the concepts and syntax takes a bit of getting used to…

In /etc/fail2ban/jail.conf` there’s a section that describes various actions – look for action_, action_mw and action_mwl.  You’ll see they vary in scope, from just writing to the logfile to emailing the sysadmin (or even administrators identified in whois lookups) or automatically banning IPs from 3rd-party services like CloudFlare.

Further down is this:

# Choose default action.  To change, just override value of 'action' with the
# interpolation to the chosen action shortcut (e.g.  action_mw, action_mwl, etc) in jail.local
# globally (section [DEFAULT]) or per specific section
action = %(action_)s

In other words, you can have a single definition in /etc/fail2ban/jail.conf and reuse it in jail.local without writing it out again in full.  It will need to go in the correct [section] (or “jail”) or under [DEFAULT].

I’d recommend changing one thing at a time – many of the checks (FTP etc.) will be disabled by default anyway.

Note: your jail.local file may have the actions written out in full as well (mine did) in which case you can just manually remove the sendmail line.  Adding a duplicate action won’t produce a warning anywhere, fail2ban will just use the last one one.

But there’s no [ssh] section? Which of these “jails” do I use?

[ssh-iptables]
[ssh-tcpwrapper]
[ssh-route]
[ssh-iptables-ipset4]
[ssh-iptables-ipset6]
[ssh-iptables-ipset6]

Check fail2ban’s status to get a list of which jails it’s using, e.g.

sudo service fail2ban status
fail2ban-server (pid  9427) is running...
Status
|- Number of jail:    1
`- Jail list:    ssh-iptables

Your default jail.local will likely already have enabled=true or false lines for each jail too.

Remember to restart the service.

sudo service fail2ban restart

Checking what Fail2Ban doing now you no longer have email alerts

See the entries in /var/log/messages, such as:

Mar 21 13:41:54 myserver fail2ban.filter[3306]: INFO [ssh-iptables] Found 123.123.123.123
Mar 21 13:41:55 myserver fail2ban.filter[3306]: INFO [ssh-iptables] Found 
123.123.123.123
Mar 21 13:41:56 myserver fail2ban.filter[3306]: INFO [ssh-iptables] Found 123.123.123.123
Mar 21 13:41:57 myserver fail2ban.filter[3306]: INFO [ssh-iptables] Found 123.123.123.123
Mar 21 13:41:58 myserver fail2ban.filter[3306]: INFO [ssh-iptables] Found 123.123.123.123
Mar 21 13:41:59 myserver fail2ban.actions[3306]: NOTICE [ssh-iptables] Ban 123.123.123.123

Introduction to Let’s Encrypt (and a few tips)

Note: this isn’t a How To or Getting Started guide – be sure to use the official documentation.  Updated Sun 25 Mar 2018.

13 May 2016 – see my summary of the latest changes.

Simpler installation process

Let’s Encrypt has been packaged on newer Linux systems – you no longer need to do manual installation. E.g, on Ubuntu 16.04 (Xenial) you can now simply do:

sudo apt-get install letsencrypt

The Certbot installation page prompts you to choose your web server and operating system/version and provides tailored instructions.

What is Let’s Encrypt?

Let’s Encrypt is a free and automated Certificate Authority.  Providing you run your own virtual or dedicated Linux server, you can quickly create and renew certificates for any domains (or subdomains) hosted on it.

Why use it (and why use https)?

Let’s Encrypt is completely free, whereas the cheapest certificates I’m aware of are currently £30+VAT per year for a single (sub)domain certificate or £80+VAT/year for a wildcard (*.example.com)

Until now, that’s made it common to avoid using https:// unless absolutely necessary – e.g. you have a customer database.  As the old process, described below, often involves clients, it can add the burden of justifying to them why spending the extra money is worthwhile.

Now, you can use TLS (https://) everywhere as standard and take advantage of the features of SPDY and HTTP/2, including faster page loads on mobile and a slight boost in your search ranking.  A secure connection also prevents mobile networks or other proxy servers from altering (or breaking) your HTML, changing your cache headers or compressing images.

Getting into the habit of building sites for https:// from the very beginning makes sense as you don’t have to check all your existing pages for insecure content as you do when migrating at a later date – I’d recommend always using a self-signed certificate on your development server.

Once you’re setup and organised – that is, you’ve installed Let’s Encrypt, written and tested a standard set of Apache or Nginx settings and been through the whole process for one domain – adding certificates for subsequent sites is going to take you little more than a minute or so.  Renewing them (note Let’s Encrypt certificates only last 90 days) takes just a few seconds.

By contrast, buying certificates the conventional way is tedious and error prone. You must:

  • pay (which also generates invoices you have to enter as part of your book-keeping)
  • generate and upload a CSR
  • (at a minimum) verify an email address on the domain name (which often involves other people and testing mailboxes in advance)
  • wait for approval and certificate generation (delays of an hour or two are common whenever I do it)
  • manually copy and paste your certificate
  • identify the correct intermediate certificate and add it to your bundle
  • install everything on the sever, check the configuration, reload and hope for the best

Browser support

At the time of writing, Let’s Encrypt is still technically in beta.

Until now, certificates haven’t worked in Windows XP, but this should be resolved by late March 2016. Because of that there are some sites I’ve not migrated from paid certificates yet, and some where I’ve only forced https:// for the admin pages.  For what it’s worth, my approach has been to check analytics data for existing sites but generally just go ahead with new ones – fortunately this won’t be a problem for much longer.

Tips and Troubleshooting

Minimising downtime, keep your existing server running

Originally, it was necessary to use the Let’s Encrypt standalone webserver to generate or renew certificates, which meant turning your existing server off temporarily to free up the port.

This is no longer the the case.  The webroot plugin creates a hidden .well_known directory in your docroot and places static files there as part of an ACME challenge to confirm ownership of the domain – therefore it’s compatible with Apache, Nginx or any other web server.

Directories beginning with a period (.) are usually protected (to stop people browsing .git or .htaccess files) so a brief bit of config is necessary. Nginx example:

location ~ /.well-known {
  allow all;
}

Recommendation: create this directory manually, stick a file in it and verify you can view it in the browser. If you get 403 Forbidden:

  • make sure the directory and files have the correct owner/group and permissions (i.e. can your nginx/www-data etc. user access them?)
  • remember Nginx’s rules about location matching. Are there location blocks earlier in the config that will take precedence over this one?
  • Remember to disable password protection, e.g. with auth_basic off;
  • If your directory is mapped to an alias, rather than a relative location under docroot, try beginning the block with location ^~ instead.

Even if you’re using the standalone server, use the command line switches to save yourself time and manually respond to the prompts.

Why can’t you run the Let’s Encrypt standalone server on some other port so you can leave your web server running? Security.  Ports 80 and 443 are privileged and it would be dangerous for just any user of the server to be able to generate certificates.

Moving servers – remember to turn Certbot off on original machine

If you move a domain from one server to another, you’ll probably remember to install/configure certbot on the new machine, but then may forget to turn it off on the old one.  This doesn’t do any harm – but it does create confusion, because at some point you’ll get an error message like this:

The client lacks sufficient authorization – 404

You may then, as I have, spend a while debugging the old server, before you realise the domain is no longer even mapped to it.

Testing certificate security and compatibility

The Qualys online test is by far the most widely used at the moment.

Opinion: given the variation in the care with which people configure their security certificates, perhaps it’s time the padlock icon changed or degraded in some way to indicate a server where encryption is valid, but vulnerable (RC4 support, for example).  We have taught non-technical users that if the padlock is there, everything is fine, which is often not the case.

Choosing the right encryption protocols

It’s very hard for most people to keep up with the frequent OpenSSL security alerts (and new developments in crytography such as elliptic curve). Worse, there’s a lot of partial advice in circulation – some of it dangerously out of date – about which settings to use.

Remy van Elst has written an excellent excellent guide to maximising your SSL score. <- This is probably the most useful link in the entire post.  Bookmark this site.   He also explains many advanced topics like OCSP Stapling.

Troubleshooting OCSP Stapling

Remember, OCSP won’t work from the very first request after restarting the server, you must allow Nginx a chance to asynchronously query the upstream certificates and cache that data.  So if you’re testing with the openssl command (see below) and it fails straight away, give it a few seconds and repeat.

Similarly, if you are only protecting, say the admin pages of the site, the Qualys SSL Labs test on the https:// version of the homepage can show “OCSP Stapling: No” when you run it, if no-one has visited an https:// protected page recently.

Note the OCSP cache is per Nginx worker, with no sharing between processes.

openssl and SNI

Brief reminder: Server Name Indication allows one server to handle lots of certificates for different domains without needing a separate IP address for each. But it doesn’t work in Internet Explorer up to and including Windows XP (fine in other Windows XP for browsers other than IE, and IE itself supports it as of  Windows 7.)

SNI is pretty standard now, of course.   If you’re running openssl from the Linux CLI, e.g. to test the server’s OCSP response…

openssl s_client -connect example.com:443 -servername example.com -tls1 -tlsextdebug -status | more

…be aware the -servername switch is essential as it indicates which of the available certificates you want, if you don’t specify it Nginx will send back the first one it finds (typically whichever domain is at the beginning of the alphabet, it seems to use the first Listen 443; directive it finds in /etc/nginx/sites-enabled)

Monitoring certificate expiry

Let’s Encrypt will email you shortly before a certificate expires, but if you’re a Nagios user you can test validity yourself using the check_ssl_cert plugin.  Example config:

define service {
    service_description     Certificate - https://wturrell.co.uk
    host_name               turtle2
    check_command           check_ssl_cert!-H turtle2.wturrell.co.uk -n wturrell.co.uk -A -c 7
    use                     daily
}

where -c is the number of days before expiry a critical alert is issued, and daily is a service with a normal_check_interval of 1440 (1440 mins = 24 hrs)

Test output:

SSL_CERT OK - X.509 certificate for 'wturrell.co.uk' from 'AlphaSSL CA - SHA256 - G2' valid until Jul 16 07:25:54 2019 GMT (expires in 1231 days)

(the example I’ve given is a commercially bought certificate, hence the long duration.)

As per the documentation, Let’s Encrypt say they’ll be streamlining the renewal process, and provide an example of a script you could use in the meantime.  I’d still advocate taking responsibility for your own monitoring if you can.

Delete unwanted/old certificates

e.g. if you no longer use a domain, say it was a test domain before the site went live, there’s no longer any need to delete files manually from the various /etc/letsencrypt directories.  First:

cerbot-auto delete

…then choose the certificate you want to remove from the list.

Troubleshooting Let’s Encrypt error messages

The client sent an unacceptable anti-replay nonce :: JWS has invalid anti-replay nonce

Run it again until it works (this was a bug earlier on due to a problem with Let’s Encrypt’s CDN – they’ve fixed that and I haven’t seen it since.)

could not find cert file

Check /var/log/letsencrypt/letsencrypt.log for the precise problem – for example, I’d manually (but only partially) deleted an unwanted certificate and it was still looking for the remainder of that, regardless of which domain I was attempting to renew.

Let’s Encrypt WILL support Windows XP very soon

Update Wed 30 March 2016:  Windows XP browsers ARE NOW SUPPORTED – you don’t need to do anything for this to take effect other than renew your certificate and restart Apache or Nginx.

At present, Let’s Encrypt certificates don’t work on Windows XP (except for Firefox.) This is because IdenTrust, their cross-signatory, requires a certificate extension known as “NameConstraints” to prevent certificates being signed for .mil (or US military) domains.  It looks a bit like this:

Permitted=None
Excluded
     [1]Subtrees (0..Max):
          DNS Name=.mil

Unfortunately XP doesn’t understand this so the certificate breaks.

Anyway, they’ve resolved this nowbut as the certificates will still be from IdenTrust I’m unclear how exactly, the new intermediate certificate won’t have nameConstraints.

This is excellent: it’ll mean there are no real arguments against moving all your existing (paid) certificates over when – or even before – they expire, or adding https:// support to sites without it.

You will need to renew all your certificates to get WinXP support, however one of the advantages of Let’s Encrypt is that the certificates have a short lifetime (3 months), so everyone gets upgraded quickly.  They may make the lifetime shorter in future, if anything.

Note also, the service is still in beta and I would hesitate to criticise them for a problem caused by a deprecated OS, compounded by a political issue.  The rest of the product is already close to perfect.

I have published some Let’s Encrypt tips.

Fixing the MailChimp API SSL certificate

Here’s a StackOverflow answer by me for anyone who sees this error:

SSL certificate problem: unable to get local issuer certificate

…when using the MailChimp API.

The problem is due to MC using a root certificate that’s been removed from the Mozilla certificate bundle (you’re most likely to see it on Debian systems.)

This is a safer way to work around the problem that rolling all your certificates back to 2014 or disabling SSL certificate verification entirely.  You also won’t need to set cURL options or edit your php.ini.

Also covered: how to ‘ping’ the MailChimp API to check it’s working.