CVE-2014-0160, better known as the Heartbleed bug, it a critical, easily exploited, and widespread bug in the OpenSSL library that powers many HTTPS implementations around the world.

It is a memory leak exploit that can be used to potentially expose server keys, and any amount of other private information, so, it’s hard to stress exactly how important it is that you patch your servers now!

Because it may help others, here’s what I did:

Diagnosis

First step is to confirm that your servers are vulnerable. The vulnerability exists in OpenSSL libraries newer than 0.9.8 and persists until fixed in 1.0.1g. This means it is present in both Debian stable and Ubuntu systems, among others.

You can confirm whether your systems are vulnerable with this handy python program (mirror on gist), written by Jared Stafford, which attempts to exploit this bug.

Fixing

  • Debian and Ubuntu users should apt-get update; apt-get upgrade as soon as possible, since the security team has already patched the vulnerability. Other distros are likely to do the same, and I wouldn’t be surprised if patches were already released.
  • If your distro hasn’t patched yet, or you’ve compiled your own code, you should update to 1.0.1g now, or recompile your current OpenSSL binary with the -DOPENSSL_NO_HEARTBEATS. option set.

Thankfully, I run Debian, so my fix was easy. However, it is important to highlight that after updating the library you must RESTART any services that make use of it, notably Apache in my case.

This seems obvious, but until you do this your server is still vulnerable, so it’s worth underlining, and it caught me out!

Verification

Finally, it is important to verify your fix (how I spotted the restart requirement!). Use the same python tool above, and you should see something like:

marcus@dushka:~$ python ssltest.py marcus-povey.co.uk
Connecting...
Sending Client Hello...
Waiting for Server Hello...
 ... received message: type = 22, ver = 0302, length = 58
 ... received message: type = 22, ver = 0302, length = 3527
 ... received message: type = 22, ver = 0302, length = 781
 ... received message: type = 22, ver = 0302, length = 4
Sending heartbeat request...
Unexpected EOF receiving record header - server closed connection
No heartbeat response received, server likely not vulnerable

Happy patching!

Update: seems that lots of people must be after that python script, as we seem to have killed the poor guy’s server. I’ve stuck a copy of it on Gist, hopefully Jared won’t mind!

GCHQoogle: so much for "Don't be evil"Given what we now know about the mass surveillance, and attack on the infrastructure of the internet, conducted by Britain’s GCHQ and America’s NSA (as well as their Chinese, Russian, German, etc counterparts).

Given that we now know, for a fact, that almost every byte of non-encrypted traffic is recorded and analysed, shouldn’t we now make a concerted effort to finally deprecate vanilla HTTP in favour of HTTP over TLS (HTTPS)?

When you use HTTP, it is a trivial matter for an attacker to see the content of the pages you visit, when, and how often you visit them. When using HTTP, there is also no guarantee that the content of the page hasn’t been modified without your knowledge, exposing you to all kinds of attacks.

Encryption, by and large, removes these problems, as well as massively increasing the cost of mass surveillance. Is it not time for all of us, as well as standards organisation like the IETF, push to make HTTPS the default? Even during my time I’ve seen insecure protocols like telnet and FTP go from widespread use to being almost completely replaced by secure alternatives (ssh and scp), so could we not do the same with HTTP?

Certificate authorities

Ok, there is one big difference between HTTPS and ssh (ok, many many, but one I care about here), and that is that HTTPS relies on certificate authorities. These are necessary in order to distribute trust, so that browsers can know to automatically accept a certificate and verify the server it is connecting to is who it says it is.

This is much nicer for the average user than, say, manually verifying the server’s fingerprint (as you have to do with SSH), but comes with some pretty serious problems if we were to make it default:

  • Every site owner would have to get a certificate, and these can only be obtained by a certificate authority if you don’t want browsers to pop up a big red warning, meaning we further bake these guys in to the Internet’s DNA.
  • Certificate authorities can be directly pressured by governments, so, a government attacker could MITM you on a secure connection and present you with a certificate that your browser accepts as valid, and so will give you no warning (of course, this is much more costly than the blanked mass surveillance that is currently going on).
  • Getting a certificate either costs money, and/or has restrictions placed on their use (for example, no commercial use, in the case of StartCom). This is really bad, since it essentially requires permission from a third party to launch a site.

It is this last causes me most concern, since it essentially provides an easy way of suppressing minority views.

Imagine that we lived in a world where HTTP had been deprecated, and browsers no longer supported unencrypted HTTP, or could, but you had to request it specifically (essentially the reverse of what we currently have). You wanted to launch a site that expressed a minority view – perhaps you were critical of your government, or you wanted to leak some information about crimes being committed, is it not inconceivable that you could have trouble obtaining a certificate? Given that certificate authorities are companies who worry about their bottom line, and are a convenient point for the bad guys to apply pressure?

If you couldn’t get a certificate in this environment, it could dramatically reduce the audience that would see your site.

So, perhaps before we move to deprecate HTTP, we must first find a better way than certificate authorities to distribute trust? How could we accomplish this? Perhaps we could take advantage of the fact that most people’s browsers automatically update, and so we could distribute browsers with expected certificates for sites hard coded into them (giving an added advantage that we could pin certificates)?

Anyway, its complicated, and I’m thinking aloud here… what are your thoughts?

WordPress, the popular blogging software written by Automattic, has a problem with SSL self signed certificates. Basically, they don’t work well in any of their newer software products or services.

In order to post an update, I must first log into my blog. This requires me entering a username and password into a login box in the usual way. By default, WordPress does not use the secure HTTPS protocol for this, instead it sends this password in the clear over HTTP.

This is not good, so I, like many others, force WordPress to carry out login and administration functions over HTTPS. This is relatively straightforward, and well documented in WordPress’ own documentation, but requires a SSL certificate.

You can obtain a SSL certificate in one of two ways. Either you pay for a third party issuer to give you one (which has the benefit of not triggering a warning in the browser), or you generate one yourself – a so called “Self Signed” certificate.

Self signed certificates are perfectly valid, but browsers will display a warning on sites which use them. A problem if you’re running a public facing service, but not if it’s just for your own private blog, and crucially the traffic is still encrypted.

The Problem

Unfortunately WordPress don’t seem to like self signed certificates.

The iOS WordPress client once worked fine with self signed certificates, but this functionality was removed in an update a few months ago. Attempts to connect now display an error about the certificate’s self signed status, but unlike all browsers, will not give you the option to proceed.

Jetpack, which is now replacing much of the functionality previously provided by separate WordPress plugins (most importantly WordPress stats), is completely broken.

When you attempt to activate the plugin, Jetpack complains about being unable to communicate with the site with the following error:

Error Details: The Jetpack server was unable to communicate with your site [IXR -32300: transport error: http_request_failed SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed]

There is no way to bypass this, since the cURL error originates on the Jetpack servers and would require a code change at their end to allow the self signed certificate.

WordPress remain tight-lipped

I am not alone in encountering these problems, but so far attempts to contact WordPress/Automattic for support by various mechanisms have all gone unanswered.

It is a legitimate point of view that certificate failures caused by self signed certificates should be a fatal. Personally, I think providing a mechanism to bypass these errors for those who know what they’re doing, is a better solution, but making it fatal is a legitimate point of view from a security standpoint.

I could resolve this issue by buying a certificate, although I have a number of good reasons, some financial and some technical, for why I have not yet done so. If Automattic were to point blank refuse to support self signed certificates in their products then I would have to find a way of making it work.

I also accept the possibility that I could have made a mistake in configuration, although I’m not sure what this could possibly be, and it is only Automattic products that are having issue.

I accept all this, however all requests for support in forum threads and direct, from myself and others, go unanswered. Bug reports for the iOS client are months old and are ignored. Similarly, direct support requests to Jetpack go unanswered.

Automattic: If self certified certificates are a feature that just won’t be supported, then please communicate with me and your other users, or at least update your FAQ. If you think I’ve made a configuration error then please say so. Please communicate, because this silence is infuriating!

Update 20/11/12: After much chasing around I’ve got a response, about JetPack at least. Seems that not allowing self signed certificates was originally a design decision (a clearer error message would have been nice!), however this decision has been re-thought and it is now seen as a bug. There is currently no time-scale as to when the issue will be addressed.