So what’s been happening?

At the beginning of the month there was the international Elgg Conference held down in Brighton. This went very well, and many thanks once again to the folks down at Brighton Uni for hosting the event and generally making the whole thing possible.

We had a good number of delegates turning up, and I spent most of the day on my feet talking and taking photos.

I gave a short presentation on OpenDD and network federalisation (as well as a bit of future gazing). I’ll post the video soon, but the slides are here. For all those who have emailed me about it, I will get back to you but I’ve been a bit snowed under!

Secondly, we released Elgg 1.2.

This is basically a point release and addresses many of the issues raised with 1.1 including speed and scailability. In terms of page loads, the average page (on a correctly configured server) now comes back in 0.7 to 2 seconds (according to yslow).

Now in the run up to Christmas, I intend to indulge myself and have a look at some ideas I’ve had but hadn’t had time to play with… but more on that later.

There are technologies around that are all working on bits of the puzzle; OpenID, OAuth etc. I think OpenDD solves a key part of the problem by providing a way to link between and move arbitrary data about in a generic and easily understandable way.

In my last post I illustrated the syndication usecase for OpenDD; how you can subscribe to a feed for a user and get a list of their activity in the form of a river. One important thing that I forgot to mention of course is there is absolutely no reason why you couldn’t use the same technique to subscribe to and receive updates about any resource.

Another powerful use case for OpenDD is its use in a live push context, and that is what I’m going to write a bit about today. Now we have already touched on the fact that OpenDD components are all atomic; they do not require a great deal of context or state around them, so for example if you establish a relationship between two entities you do not need to include the complete description of the entities in question – just the relationship.

In the latest specifications I have introduced the concept of being able to unset and remove things, so as well as creating a relationship it is also possible to remove it. This turns OpenDD into quite a powerful tool in your arsenal.

If you take Elgg as an example, Elgg has an events system which we have touched on in previous articles and as well as providing a syndication of an events log in the form of a river, we can also look at pushing those events as they happen out to interesting parties as a sequence of OpenDD components.

So, for example, you could request to be notified when a user joins a group or when something changes on a file (or indeed any other resource) and this can then be pushed out to you in a way that is simple to understand – and again, because we have the concept of an UUID, it isn’t necessary to have the complete context in order for the message to be useful. Everything else can be worked out.

So, using OpenDD, it is now possible to subscribe to users and resources on different networks and receive a feed of updates, but it is also now possible to push messages between networks in an open way that is understood by both platforms.

One of the cool things you can do with that is for example Sharing. In one network you could create a resource, for example a photo of you at a party or a URL you were interested in. Then you could select between the friends on the network you are in but also from friends elsewhere to share the resource with.

The user on the foreign network would then receive an OpenDD push update message saying that you have created a resource and have shared it with them (or rather the platform would receive and interpret it), and because users and the resource are all addressed by UUID the user would be able to see this seamlessly on their own network.

This feature – made popular by the facebook “mini-feed” – has become one of the things that users expect from any platform they use and is rather compelling.

What if you want to see what they have been doing on other networks? Would it not be a pretty neat thing to be able to subscribe to their activity in order to receive updates about what they have been up to in a similar way as you would be able to subscribe to their blog via RSS?

This turns out to be a simple yet powerful use case for OpenDD, and this is how we’re doing it in the new version of Elgg.

I talked a bit about how the river was implemented in Elgg 1.0 in previous articles. In a nutshell we have a system log which stores events as they happen in terms of a simple relationship, e.g. “User X CREATED BlogPost”, “User X UPDATED profile”, etc.

Once you have that information it becomes very easy to mine that for extra information and turn it into a River event. Essentially, a plugin would create a view on the specific event and entity and then be able to render it as a river item and provide new views on existing data.

If you remember, I also talked about how I changed the river code to use this thing called ElggRiverStatement, which lets you construct the river in a much more linguistic way and also provides all the information needed by the river views.

Creating a “Friends activity elsewhere” syndication view becomes quite easy. Essentially, all you need to do is export the system log for a user, which you can do very easily using OpenDD as a sequence of OpenDD Relationships. This is because “User X CREATED BlogPost” can be thought of in terms of a relationship.

Once this is done, an aggregation client can retrieve the feeds of all the friends that you’ve signed up to and then sort the relationships by published data, and then – because everything has a UUID – you can then drill down and pull the extra information required – details about the user who initiated the action, the object the event was performed on, metadata etc.

This essentially means you can construct a succession of ElggRiverStatement objects out of an OpenDD feed. This is quite cool, because it means you can take those objects and inject them into the already existing river views and get a river entry back. This means that for absolutely zero extra work, every plugin has the ability to render a “Friend elsewhere” event… and it becomes seamless!

Additionally, you can look to digging further and getting extra information about the entities involved and actually be able to see the entity in question – whether by linking to the entity elsewhere or using the information you have about it to render it locally.

This is quite a simple example, but it shows some of the power of OpenDD. Additionally, I should point out that since OpenDD is atomic, you don’t ever have to get all the information in one go!

I would therefore like to introduce the first draft of OpenDD over Atom, submitted for discussion.

In a nutshell, OpenDD over Atom makes use of Atom’s ability to embed other formats within the content tag and simply wraps a OpenDD element within an Atom entry with some additional data from the ODD element used in the Atom “envelope”

This seemed like the simplest solution, but let me know what you think!

Import example

The import function accepts an OpenDD XML document and returns an ODDDocument object, which can then be iterated through to obtain access to the ODD* classes.

The XML file:

<odd version="1.0" generated="Fri, 09 May 2008 18:53:37 +0100">
<entity uuid="http://www.example.com/a" class="foo" subclass="http://www.example.com/b" />
<metadata uuid="http://www.example.com/c" entity_uuid="http://www.example.com/a" name="Squid" >paper</metadata>
<relationship uuid1="http://www.example.com/a" type="hates" uuid2="http://www.example.com/b" />
</odd>

Example code to process it:

$doc = ODD_Import($in);

foreach ($doc as $element)
print_r($element); // Just echo, but you should probably do something meaningful.

Export example

Export involves constructing an ODDDocument object and populating it with various classes, each one representing one of the OpenDD elements.

This object is then serialised.

Here is some example code:

$out = new ODDDocument(); // Create new document

$entity = new ODDEntity("http://www.example.com/a", "foo", "http://www.example.com/b");
$out->addElement($entity);

$meta = new ODDMetaData("http://www.example.com/c", "http://www.example.com/a", "Squid", "paper");
$out->addElement($meta);

$rel = new ODDRelationship("http://www.example.com/a", "hates", "http://www.example.com/b");
$out->addElement($rel);

echo ODD_Export($out); // Export


When I get a chance I’ll upload some libraries in other languages, but if you feel inclined then feel free to implement your own!

Marc made an excellent blog post on the subject which clearly illustrates that he completely groks what we are trying to do.

We have some exciting times ahead of us, and are going to try and move this forward as fast as we can.

As always, if you want to be a part of it then please get involved!

I will blog a bit more about it when I can, its been a bit mad so this is the first chance I’ve actually had to sit down at the computer since getting here (and I’m only able to do that because the combination of jet lag and the steak the size of my head I had for dinner last night has given me a case of insomnia).

I just wanted to share with you a thought that me and Ben had while enjoying the Californian sunshine inbetween meetings, namely a way to link ODD documents with OpenID.

For descovery of ODD documents, I was planning to use the meta / link approach similar to the way RSS is picked up. Now, it occurs to me that if we modify the spec slightly to say that a UUID should point to a page that can either be an ODD representation of the thing that it’s referring to or knows where to get it – i.e. has the appropriate header tag pointing to the url – then it becomes a trivial matter to turn a UUID into an OpenID URL.

Potentially quite useful.

Since the last release of the draft I’ve done a fair amount of work to simplify the format even further; simplifying terminology, clearing up some inconsistencies and dropping namespaces altogether.

You’ll notice that we still don’t define any terms. As Ben touched on in a recent post, we decided to not confuse the format by trying to tie it to any one application, while keeping it as easy as possibly to actually use. I’ll cover this in more detail a bit later…

So, lets talk about how I’m using ODD to implement full data import and export in the upcoming release of Elgg – Elgg 1.

For those who don’t already know, Elgg is an Open source social networking application engine. The previous version has been downloaded over 100K times, and Import and export was one of the most frequently requested enhancements.

Export

Export was a fairly trivial matter. The new version of Elgg employs a flexible event system, so all I had to do was trigger an “export” event.

This event is passed a GUID – an identifier identifying the thing you are exporting, and elements of the system (and thirdparty plugins) can listen for this event and react accordingly.

The event is essentially asking all parts of the Elgg application – core and plugins – “Tell me all you know about X”. The export listens to the answers and converts it into an ODD document that looks something like this:

<odd version="1.0" generated="Wed, 30 Apr 2008 22:21:55 +0100">


<entity uuid="http://example.com/odd/78/" class="object" subclass="blog" published="Fri, 18 Apr 2008 11:45:50 +0100" />


<metadata uuid="http://example.com/odd/78/attr/owner_uuid/" entity_uuid="http://example.com/odd/78/" name="owner_uuid" published="Fri, 18 Apr 2008 11:45:50 +0100" >http://example.com/odd/77/</metadata>


<metadata uuid="http://example.com/odd/78/attr/title/" entity_uuid="http://example.com/odd/78/" name="title" published="Fri, 18 Apr 2008 11:45:50 +0100" >test</metadata>


<metadata uuid="http://example.com/odd/78/attr/description/" entity_uuid="http://example.com/odd/78/" name="post" published="Fri, 18 Apr 2008 11:45:50 +0100" >First post</metadata>


<metadata uuid="http://example.com/odd/78/metadata/35/" entity_uuid="http://hexample.com/odd/78/" name="tags" type="metadata" owner_uuid="http://example.com/odd/77/" published="Fri, 18 Apr 2008 11:45:50 +0100" >wibble</metadata>


</odd>


Here we see an entity (in this case a blog post), and some details about it (the metadata).

Import

Import is traditionally the more complicated part of the equation. ODD is trivial to parse, each tag is atomic and represents exactly one thing, this is a big advantage from the point of view of anyone implementing a reader for it since it makes the whole thing pretty much stateless.

ODD tags arrive, whether as a file to import or as a live feed, and an event is triggered. This event passes around the tag and essentially asks the question “Does anyone know how to handle this?”.

The stateless nature of ODD of course meaning that you don’t have to process the entire file, making it a trivial matter to implement a reader using a SAX parser.

That just about covers it, I’ll be posting some example code in a few days (workload permitting) so hopefully people can start getting stuck in. If you want to get involved in development, please head over to the ODD group.

A final note: I will be in San-Francisco all next week, so if you are in the bay area and feel like having a chat about ODD or Elgg, then please get in touch!

Ben discusses ODD and the other data portability formats in the area and explains where ODD fits in.

Enjoy!

Its been a while since I have done anything remotely like public speaking so I was rather glad that the event had plenty of “Free as in Beer” beer.

Those of you who are interested, I have uploaded a PDF version of my slides here.

ODD has generated a fair amount of comment. One commenter suggested that we were re-inventing the wheel somewhat.

Maybe that is partially true – there are other data portability formats available, RDF for example or SIOC (apparently pronounced “Shock”, although I’m not entirely sure how). The point we are making with ODD is that powerful as many of these formats are, they are just too complicated and in many cases ambiguous, and for those reasons are not going to see widespread adoption.

Tracked vehicles are very powerful and versatile, but sometimes you just need a bike.

RSS is a good example of what I’m talking about. RSS is nice and simple, and as a result has seen widespread industry adoption. Crucially too there are many applications that consume RSS as well as just produce it, which is something not many other formats can boast.

Our view is that while many of these formats are academically brilliant and conceptually very clever, but they are just too complicated.