mysql-logo3 So, the other week, I migrated all my sites over to a new server. This was accomplished with minimum fuss, using lots of rsync magic and juggling of DNS ttls.

The part of the migration I imagined would be the most complicated, moving several tens of MySQL databases running on the old server to the new, turned out to be pretty straightforward, and essentially completed with one command. I thought others might find it handy to know how…

First, I brought down Apache on both servers, so I could be sure that nobody was going to try and write to the database. This may not really be required, since mysqldump can handle dumping live databases consistently, for MyISAM at least, but in never hurts to be paranoid.

Next, I needed to move all the databases, together with their access permissions and users (so the full MySQL schema) to the other server. One way to do that is to copy the whole /var/lib/mysql directory over. However, I had a lot of cruft in there (old bin-logs etc), plus there were a number of articles suggesting that the straight binary copy had a number of issues, especially for mixed storage engine environments. So, I opted for the mysqldump method.

Traditionally, this takes a lot of SCPing. Here’s how to do it with one command, using the magic of Unix pipes:

mysqldump -u root -pPASSWORD --all-databases | ssh USER@NEW.HOST.COM 'cat - | mysql -u root -pPASSWORD'

Boom. This ran surprisingly quickly for me, and you can of course just as easily use this method to transfer a single database.

Three gotchas:

  • If the link dies, you need to start again, so don’t do this over a flakey connection, and I suggest you run the command in a screen if the first server isn’t localhost.
  • You need to restart the database server on the target machine for the new users and privileges to come into effect.
  • On Debian, you may see an error along the lines of:

    Got error: 1045: Access denied for user ‘debian-sys-maint’@’localhost’ (using password: YES) when trying to connect

    Fix this by executing the command:

    GRANT ALL PRIVILEGES ON *.* TO 'debian-sys-maint'@'localhost' IDENTIFIED BY 'THEPASSWORD' WITH GRANT OPTION;

    Where THEPASSWORD is the password found in /etc/mysql/debian.cnf

One final note: this will only work if both servers are the same major version number. I was moving between two Debian 6.0 installs, so YMMV.

Happy Easter!

If you’re anything like me, you often find yourself needing to add create and modified timestamps to data stored in a MySQL database. Typically, I’d do this programmatically from within the application without giving it a second thought, wrapping up time(); in an SQL statement and firing it over.

Recently, while doing some development work for one of my clients, I was given the requirement to add Created and Modified timestamps to a whole bunch of existing data tables. These tables were referenced by hundreds of different MySQL queries, and to complicate matters further, we were in the middle of migrating the code over to a new database library.

I didn’t much fancy rewriting all of the SQL queries, potentially twice, just to add a couple of timestamps, so modifying code wasn’t much of an option. Thankfully, a couple of MySQL’s internal features came to the rescue.

Adding a ModifiedTime

Adding a modified timestamp to a table is the most straight forward. All your have to do is create the field of type TIMESTAMP, and by default, MySQL will automatically update the field when the row is modified.

There are a couple of things to be aware of:

  • While you can have multiple TIMESTAMP fields in a row, only one of these can be automatically updated with the current time on update.
  • If your UPDATE query contains a value for your ModifiedTime field, this value will be used.

So, to add your modified timestamp field to an existing table, all you need is:

ALTER TABLE my_table ADD ModifiedTime TIMESTAMP;

Adding a CreatedTime

Adding a CreateTime value is a little more involved.

On the latest versions of MySQL it is apparently possible to create a DateTime field with a default value of CURRENT_TIMESTAMP. This wasn’t an option for me as I was having to support a somewhat older version, besides, even on the newer versions of MySQL it is not possible to have more than one field using CURRENT_TIMESTAMP, which of course we are in order to get ModifiedTime working.

So, in order to get a created timestamp, firstly we must add a DATETIME field to the table.

ALTER TABLE my_table ADD CreatedTime datetime NOT NULL;

Note, that this must be created as NOT NULL in order for the next part to work (this is because setting NOT NULL forces an automatic all zeros default).

Next, we must create a trigger, which will automatically be fired when we insert a value into our table and set the created timestamp.

DELIMITER //
DROP TRIGGER IF EXISTS `my_table_insert_trigger`//
CREATE TRIGGER `my_table_insert_trigger`
BEFORE INSERT ON `my_table`
FOR EACH ROW
BEGIN
IF NEW.CreatedTime = '0000-00-00 00:00:00' THEN
SET NEW.CreatedTime = NOW();
END IF;
END;//
DELIMITER ;

Now, when you insert a value into the table, this trigger will fire and, if you’ve not provided a CreatedTime field in your insert query, it will be set to the current time stamp.

Conclusion

Since all the queries in the application code specified the columns they were updating on insert, I was able to use this method to add created and modified time stamp fields to all the existing object tables in the database, without needing to modify any of the existing application code.

This simplified my life greatly, but also suggests to me that this method might be somewhat more efficient than the methods I’d previously used.

As I’ve blogged before, IFTTT.com (short for “If This Then That”) is a popular service which lets you trigger actions based on certain events that occur around the internet.

One of the most requested features, by programmers at least, is to add WebHooks support. Webhooks are a very simple way of pinging API information about the internet between web services. It uses JSON to send an arbitrary payload over HTTP via a POST request to a specified endpoint. This is about as simple as you can get, which is of course the beauty of it.

Support for Webhooks is an obvious extension to IFTTT, and would allow people to build on the service, connecting together more than just the hand picked menu of channels on the IFTTT dashboard. Quite why this is so slow coming is a mystery; some have speculated that it was a business decision on their part, others that it is hard to build a slick interface for something of such a highly technical nature, others that they simply haven’t gotten around to it just yet.

Free Software to the Rescue!

Until IFTTT implement a WebHooks channel, you can use the following workaround.

Abhay Rana, in a project over on GitHub, has built a some code which provides a WebHooks bridge for IFTTT and other services. I have extend to add some extra functionality you might find useful.

Currently, the IFTTT wordpress channel uses that software’s RPC endpoint to make posts, so the software works by providing a little bit of middleware, that you install on an internet facing machine, which pretends to be an installation of WordPress. You then point the wordpress IFTTT channel at this installation, passing it some special parameters.

You specify the final endpoint URL as a tag, and by default the contents of the post, title, and categories get JSON encoded and relayed to this endpoint.

My extensions

Different Webhook endpoints need different fields, however, so my extension lets you provide service specific plugins. These plugins can manipulate the data sent to the upstream endpoint further, providing different fields and encoding options. This lets you support multiple webhook endpoints from a single installation, and without having to set up multiple IFTTT accounts.

To use, create the appropriate plugin in your installation’s plugins directory containing your extension of the Plugin class, then pass a special category “plugin:NameOfClass“.

I have included an example class and a JSON payload class. The latter assumes that the post body is valid JSON, validates it, and then sends it to the upstream endpoint. This lets you send specially crafted messages to upstream webhook endpoints.

My extension also includes much more debug logging, as well as some extra validation code and graceful failures.

Why?

IFTTT is a fantastically useful service, but unfortunately you are limited to using the services they choose to (or have time to) write connectors for. I find this limiting, and at times frustrating, since as well as excluding some great existing services, it places limits on my ability to hack my own stuff together!

Previously, I’d often used twitter to glue things together. However, since Twitter are currently making concerted efforts to turn their service into just another ad serving platform, rather than the communication platform and messaging service it was growing into, it was necessary to come up with an alternative method.

WebHooks seem a better solution anyway.

Thanks again to Abhay Rana for the original code, and I hope people find my additions useful!

» Visit the project on Github…