I have previously talked about speeding up your site by using Squid as a reverse proxy to cache served pages. This is a great thing to do, but presents a problem now that all sites are moving over to HTTPS, since for various technical reasons reverse proxies can’t really handle HTTPS.

These days the standard way of doing this seems to be using Varnish as a cache, and Squid seems to be a little “old hat”, however I have several client estates which were set up before Varnish came on the scene, so I needed a solution I could get up and running very quickly.

Terminating HTTPS

Thankfully, the solution is very similar whatever reverse proxy you’re using. The solution is simple, you need to install something that terminates and handles the HTTPS session before sending it to your proxy. The simplest way to do this is to install NGINX and configure it to handle HTTPS sessions.

1) Disable Apache’s handling of HTTPS (if you’ve got an existing, un-cached, HTTPS server).

2) Install the basic nginx apt-get install nginx-light

3) Configure nginx to forward to your proxy (which you have previously configured to listen on port 80)

server {
        listen 443 ssl;
        server_name www.example.com;
    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

        location / {
            proxy_pass http://127.0.0.1:80;
            proxy_set_header X-Real-IP  $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto https;
            proxy_set_header X-Forwarded-Port 443;
            proxy_set_header Host $host;
        }
}

After restarting nginx, you should be able to see https requests coming in on your squid proxy logs.

Gotchas

The biggest gotcha that you’re going to hit is that if you’re checking whether a request is HTTPS in your app (e.g. for automatically forwarding from insecure to secure), you’re not going to be able to use the standard protocol checks. The reason being is that HTTPS is being terminated by nginx, so by the time the session hits your app, it will not be seen as secure!

To perform such a test, you’re instead going to have to check for the X-Forwarded-Proto header instead ($_SERVER['HTTP_X_FORWARDED_PROTO'] in PHP).

If you run a web server and you take a look at the logs, you will likely have seen something like this appearing:

xxx.xxx.xxx.xxx - - [01/May/2013:18:32:36 +0100] "GET /w00tw00t.at.ISC.SANS.DFind:)
HTTP/1.1" 400 320 "-" "-"

This is the product of a tool called w00tw00t, which is used by nefarious script kiddies to probe and attempt to compromise your server. If your server is up-to-date, there is probably not too much to worry about (without wanting to jinx it), however since a strength in depth approach is always the best plan when it comes to security, it is probably a good idea if we could deploy some additional countermeasures.

A common tactic is to use a tool like fail2ban to monitor your logs and then firewall off the offenders IP address, and there are filters out there to do this.

However, like a lot of people, I use the caching proxy Squid, in reverse proxy mode, to help handle high load on a web server. Since, in this configuration, Apache (and therefore fail2ban’s standard w00tw00t rules) will see these requests as coming from the cache machine, we need to take another approach.

One option is to modify the Apache log format to use the X-Forwarded-For variable instead (details of how can be found here), and thus preserving the original IP address in the logs. However, this would require me to modify a number of vhosts, and it seemed simpler to monitor the one squid access log.

So, I wrote a quick fail2ban filter to catch w00tw00t scans and block the offending IP address.

In jail.local

[squid-w00tw00t]
enabled = true
filter = squid-w00tw00t
port = all
logpath = /var/log/squid/access.log
maxretry = 1

In filter.d/squid-w00tw00t.conf

# Fail2Ban configuration file to catch w00tw00t scans on squid reverse proxy settings
#
# Author: Marcus Povey
#

[INCLUDES]

# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf

[Definition]

_daemon = squid

# Option: failregex
# Notes.: regex to match the password failures messages in the logfile. The
# host must be matched by a group named "host". The tag "" can
# be used for standard IP/hostname matching and is only an alias for
# (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
# Values: TEXT
#
failregex = <HOST> TCP_.*http.*/(w00tw00t|wootwoot|WootWoot|WooTWooT).*$

# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =

» Visit the project on Github…

The server-status page gives you a wealth of information about your apache server, and among other things it is necessary for the Apache munin plugin to work. However, by default it exposes sensitive data when run behind a squid reverse proxy.

In order to lock this down, you need to modify your /etc/squid/squid.conf file slightly…

acl no_stats urlpath_regex /server-status
http_access deny no_stats

This defines an ACL that will prevent squid from giving access to any request ending in /server-status. Because, in a reverse proxy configuration, it is squid that is hit when a client request a page from port 80, this prevents public access.

You’re not quite done though, because this setting assumes that your real web server (sitting behind the squid proxy on a different port) is NOT publicly visible. This can be achieved by placing it behind the firewall.

Munin

If you’re using munin to monitor your apache processes, you’ll need to make a small modification to the /etc/munin/plugin-conf.d/munin-node file, add/modify the following:

[apache_*]
env.url http://127.0.0.1:%d/server-status?auto
env.ports [apache server port]

This will tell the apache plugins where the server status page is, and what port the real apache instance is running on.