Category Archives: Technology

Posted on
by

By default, the standard LAMP (Linux Apache Mysql Php/Perl/Python) stack doesn’t come particularly well optimised for handling more than a trivial amount of load. For most people this isn’t a problem, either they’re running on a large enough server or their traffic is at a level that they never hit against the limits.

Anyway, I’ve hit against these limits on a number of occasions now, and while there are many good articles out there on the subject, I thought I’d write down my notes. For my own sake as much as anything else…

Apache

Apache’s default configuration on most Linux distributions is not the most helpful, and you’re goal here is to do everything possible to avoid the server having to hit the swap and start thrashing.

  • MaxClients – The important one. If this is too high, apache will merrily spawn new servers to handle new requests, which is great until the server runs out of memory and dies. Rule of thumb:

    MaxClients = (Memory - other running stuff) / average size of apache process.

    If you’re serving dynamic PHP pages or pull a lot of data from databases etc the amount of memory a process takes up can quickly balloon to a very large value – sometimes as much as 15-20mb in size. Over time all running Apache processes will be the size of your largest script.

  • MaxRequestsPerChild – Setting this to a non-zero value will cause these large spawned processes to eventually die and free their memory. Generally this is a good thing, but set the value fairly high, say a few thousand.
  • KeepAliveTimeout – By default, apache keeps connections open for 15 seconds waiting for subsequent connections from the same client. This can cause processes to sit around, eating up memory and resources which could be used for incoming requests.
  • KeepAlive – If your average number of requests from different IP addresses is greater than the value of MaxClients (as it is in most typical thundering herd slashdottings), strongly consider turning this off.

Caching

  • Squid – Squid Reverse Proxy sits on your server and caches requests, turning expensive dynamic pages into simple static ones, meaning that at periods of high load, requests never need to touch apache. Configuration seems complex at first, but all that is really required is to run apache on a different port (say 8080), run squid on port 80 and configure apache as a caching peer, e.g.


    http_port 80 accel defaultsite=www.mysite.com vhost
    cache_peer 127.0.0.1 parent 81 0 no-query originserver login=PASS name=myAccel

    One gotcha I found is that you have to name domains you’ll accept proxying for, otherwise you’ll get a bunch of Access Denied errors, meaning that in a vhost environment with multiple domains this can be a bit fiddly.

    A workaround is to specify an ACL with the toplevel domains specified, e.g.

    acl our_sites dstdomain .uk .com .net .org

    http_access allow our_sites
    cache_peer_access myAccel allow our_sites

  • PHP code cache – Opcode caching can boost performance by caching compiled PHP. There are a number out there, but I use xcache, purely because it was easily apt-gettable.

PHP

It goes without saying that you’d probably want to make your website code as optimal as possible, but don’t spend too much energy over this – there are lower hanging fruit, and as a rule of thumb memory and CPU is cheap when compared to developer resources.

That said, PHP is full of happy little gotchas, so…

  • Chunk output – If your script makes use of output buffering (which Elgg does, and a number of other frameworks do too), be sure that when you finally echo the buffer you do it in chunks.

    Turns out (and this bit us on the bum when building Elgg) there is a bug/feature/interaction between Apache and PHP (some internal buffer that gets burst or something) which can add multiple seconds onto a page delivery if you attempt to output large blocks of data all at once.

  • Avoid calling array_merge in a loop – When profiling Elgg some time ago I discovered that array_merge was (and I believe still is) horrifically expensive. The function does a lot of validation which in most cases isn’t necessary and calling it in a loop is ruinous. Consider using the “+” operator instead.
  • Profile – Profile your code using x-debug, find out where the bottlenecks are, you’d be surprised what is expensive and what isn’t (see the previous point).

Non-exclusive list, hope it helps!

Posted on
by

Once upon a time it used to be easy to find free wifi. If you needed to check your email or do a bit of work while on the go it just used to be a matter of popping into the nearest coffee shop.

But increasingly these pools of signal seem to be drying up, sometimes being replaced by costly pay-for-access hotspots, sometimes vanishing entirely.

The few free (as in beer) access points still around are increasingly not free (as in freedom) to access. Many force you to enter personal details, which is, at best, time consuming if you just want to read your email while drinking your coffee, and, at worst, downright invasive. Others, like the one at my former favourite office away from home, are now requiring you to ask for an access token – again, not something you’re going to do if in a hurry or if you are just there for a coffee.

Some coffee shops (Costa I’m looking at you) have even gone to the point of requiring you to join their rewards club and / or turning off all the power points so you can’t recharge.

All of these measures give a sense that they don’t really want their customers to use the wifi they’re providing.

What could be the reasons behind this trend I wonder? Here are a few off the top of my head…

People aren’t drinking enough coffee

One possibility in the above coffee shop example is that people are setting up with their laptops, monopolising a table, and not drinking enough coffee. I think this is probably because they are engrossed in what they are doing, or are unwilling to leave valuable laptops unattended while ordering, rather than necessarily scamming the store for the free wifi.

Regardless of the reasons it is obviously a problem, but one I’m pretty sure could be solved by having table service, and the availability of accessible wifi and readily available coffee and tasty nibbles brought to your table while working would certainly make me spend more money.

Times are hard

Thanks to government overspending, the collapse of the banking system, and a few illegal wars, times are hard and we’re all feeling the pinch. Perhaps the idea of giving away something for nothing (even if that something costs nothing to provide) may not seem as attractive a prospect as it once did.

Collecting user details seems to also be a bit of a cynical attempt to monetise the customer in new ways.

The Digital Economy Act

I think that it is more than just a coincidence that the decline in open wifi started happening after Labour’s disastrous digital economy bill was bullied through parliament by the BPI.

This bill transfers liability for anything illegal that an internet user does to the person who pays the bill, meaning that a coffee shop would be responsible for any copyright infringement performed by their customers.

This is obviously a strong disincentive for anyone wanting to run open wifi in their place of business, and runs counter to the stated aims of the bill (the real aims of course being to further entrench established monopolies, but that’s a subject for another rant blog post).

Myself, I’m thinking about just getting a Mifi on Giffgaff and giving up on the whole ubiquitous wifi concept, but that seems like admitting defeat. This doesn’t seem to be a problem elsewhere in the world, what are your thoughts?

Posted on
by

The Raspberry Pi is a tiny, solid state, and ludicrously cheap hobby ARM based computer designed in the UK (but thanks to insane UK tax laws needs to be built in China). It has a USB port, video, sound, an Ethernet port, 256MB RAM, and can run 3 distinct flavours of Linux.

Ostensibly the device was developed with the aim of getting kids to code, and as someone who grew up with the UK hobby computing scene of the 1980s and cut their programming teeth hacking games together on the ZX Spectrum, this is something I can thoroughly get behind.

The blurb from their website:

The Raspberry Pi is a credit-card sized computer that plugs into your TV and a keyboard. It’s a capable little PC which can be used for many of the things that your desktop PC does, like spreadsheets, word-processing and games. It also plays high-definition video. We want to see it being used by kids all over the world to learn programming.

Coming out of Cambridge and with a decidedly hobbyist feel to it, the Raspberry Pi could almost be the spiritual successor to the humble and much loved Speccy. If it can get more kids coding then that’s all for the good, especially if it gives the UK tech scene a much needed shot in the arm.

This is not the reason why I’m excited.

Small. Capable. CHEAP.

The Raspberry Pi is tiny, which means it can be put in tiny things. It is low power and solid state, which means it doesn’t need much juice to run (4 AA batteries will do the trick) and it can take a fair amount of abuse.

Above all, it is cheap, and this is why I’m really excited. They are certainly something you can afford to buy more of than a traditional computer, even on a modest budget. I’d go further and say that they are so cheap that they can be thought of as practically disposable general computing units… this is game changing.

The reason I am really excited about this is that all these factors combine to make them the perfect choice for the control computer for any number of appliances or devices, and it reduces the barrier to entry for the home hacker to start putting some really cool things together.

I don’t think it will be long before we start seeing countless hobbyist developed bits of hardware; from internet radios, to cheap NAS appliances, right through to remote sensor platforms, robots, drones and maybe even spacecraft. Given the number of bits of Lego sent to the edge of space recently I don’t think this is too far off!

Increasingly you don’t need to wait for a company with a massive fabrication plant to see a market for a product in order to get one, but now micro-manufacturing is something you can do in your own home. I can think of hundreds of gizmos I could spend countless afternoons building with a Raspberry Pi at the centre.

I can’t wait to get my hands on one!