I would therefore like to introduce the first draft of OpenDD over Atom, submitted for discussion.

In a nutshell, OpenDD over Atom makes use of Atom’s ability to embed other formats within the content tag and simply wraps a OpenDD element within an Atom entry with some additional data from the ODD element used in the Atom “envelope”

This seemed like the simplest solution, but let me know what you think!

Import example

The import function accepts an OpenDD XML document and returns an ODDDocument object, which can then be iterated through to obtain access to the ODD* classes.

The XML file:

<odd version="1.0" generated="Fri, 09 May 2008 18:53:37 +0100">
<entity uuid="http://www.example.com/a" class="foo" subclass="http://www.example.com/b" />
<metadata uuid="http://www.example.com/c" entity_uuid="http://www.example.com/a" name="Squid" >paper</metadata>
<relationship uuid1="http://www.example.com/a" type="hates" uuid2="http://www.example.com/b" />
</odd>

Example code to process it:

$doc = ODD_Import($in);

foreach ($doc as $element)
print_r($element); // Just echo, but you should probably do something meaningful.

Export example

Export involves constructing an ODDDocument object and populating it with various classes, each one representing one of the OpenDD elements.

This object is then serialised.

Here is some example code:

$out = new ODDDocument(); // Create new document

$entity = new ODDEntity("http://www.example.com/a", "foo", "http://www.example.com/b");
$out->addElement($entity);

$meta = new ODDMetaData("http://www.example.com/c", "http://www.example.com/a", "Squid", "paper");
$out->addElement($meta);

$rel = new ODDRelationship("http://www.example.com/a", "hates", "http://www.example.com/b");
$out->addElement($rel);

echo ODD_Export($out); // Export


When I get a chance I’ll upload some libraries in other languages, but if you feel inclined then feel free to implement your own!

Marc made an excellent blog post on the subject which clearly illustrates that he completely groks what we are trying to do.

We have some exciting times ahead of us, and are going to try and move this forward as fast as we can.

As always, if you want to be a part of it then please get involved!

I will blog a bit more about it when I can, its been a bit mad so this is the first chance I’ve actually had to sit down at the computer since getting here (and I’m only able to do that because the combination of jet lag and the steak the size of my head I had for dinner last night has given me a case of insomnia).

I just wanted to share with you a thought that me and Ben had while enjoying the Californian sunshine inbetween meetings, namely a way to link ODD documents with OpenID.

For descovery of ODD documents, I was planning to use the meta / link approach similar to the way RSS is picked up. Now, it occurs to me that if we modify the spec slightly to say that a UUID should point to a page that can either be an ODD representation of the thing that it’s referring to or knows where to get it – i.e. has the appropriate header tag pointing to the url – then it becomes a trivial matter to turn a UUID into an OpenID URL.

Potentially quite useful.

Since the last release of the draft I’ve done a fair amount of work to simplify the format even further; simplifying terminology, clearing up some inconsistencies and dropping namespaces altogether.

You’ll notice that we still don’t define any terms. As Ben touched on in a recent post, we decided to not confuse the format by trying to tie it to any one application, while keeping it as easy as possibly to actually use. I’ll cover this in more detail a bit later…

So, lets talk about how I’m using ODD to implement full data import and export in the upcoming release of Elgg – Elgg 1.

For those who don’t already know, Elgg is an Open source social networking application engine. The previous version has been downloaded over 100K times, and Import and export was one of the most frequently requested enhancements.

Export

Export was a fairly trivial matter. The new version of Elgg employs a flexible event system, so all I had to do was trigger an “export” event.

This event is passed a GUID – an identifier identifying the thing you are exporting, and elements of the system (and thirdparty plugins) can listen for this event and react accordingly.

The event is essentially asking all parts of the Elgg application – core and plugins – “Tell me all you know about X”. The export listens to the answers and converts it into an ODD document that looks something like this:

<odd version="1.0" generated="Wed, 30 Apr 2008 22:21:55 +0100">


<entity uuid="http://example.com/odd/78/" class="object" subclass="blog" published="Fri, 18 Apr 2008 11:45:50 +0100" />


<metadata uuid="http://example.com/odd/78/attr/owner_uuid/" entity_uuid="http://example.com/odd/78/" name="owner_uuid" published="Fri, 18 Apr 2008 11:45:50 +0100" >http://example.com/odd/77/</metadata>


<metadata uuid="http://example.com/odd/78/attr/title/" entity_uuid="http://example.com/odd/78/" name="title" published="Fri, 18 Apr 2008 11:45:50 +0100" >test</metadata>


<metadata uuid="http://example.com/odd/78/attr/description/" entity_uuid="http://example.com/odd/78/" name="post" published="Fri, 18 Apr 2008 11:45:50 +0100" >First post</metadata>


<metadata uuid="http://example.com/odd/78/metadata/35/" entity_uuid="http://hexample.com/odd/78/" name="tags" type="metadata" owner_uuid="http://example.com/odd/77/" published="Fri, 18 Apr 2008 11:45:50 +0100" >wibble</metadata>


</odd>


Here we see an entity (in this case a blog post), and some details about it (the metadata).

Import

Import is traditionally the more complicated part of the equation. ODD is trivial to parse, each tag is atomic and represents exactly one thing, this is a big advantage from the point of view of anyone implementing a reader for it since it makes the whole thing pretty much stateless.

ODD tags arrive, whether as a file to import or as a live feed, and an event is triggered. This event passes around the tag and essentially asks the question “Does anyone know how to handle this?”.

The stateless nature of ODD of course meaning that you don’t have to process the entire file, making it a trivial matter to implement a reader using a SAX parser.

That just about covers it, I’ll be posting some example code in a few days (workload permitting) so hopefully people can start getting stuck in. If you want to get involved in development, please head over to the ODD group.

A final note: I will be in San-Francisco all next week, so if you are in the bay area and feel like having a chat about ODD or Elgg, then please get in touch!

Ben discusses ODD and the other data portability formats in the area and explains where ODD fits in.

Enjoy!

Its been a while since I have done anything remotely like public speaking so I was rather glad that the event had plenty of “Free as in Beer” beer.

Those of you who are interested, I have uploaded a PDF version of my slides here.

ODD has generated a fair amount of comment. One commenter suggested that we were re-inventing the wheel somewhat.

Maybe that is partially true – there are other data portability formats available, RDF for example or SIOC (apparently pronounced “Shock”, although I’m not entirely sure how). The point we are making with ODD is that powerful as many of these formats are, they are just too complicated and in many cases ambiguous, and for those reasons are not going to see widespread adoption.

Tracked vehicles are very powerful and versatile, but sometimes you just need a bike.

RSS is a good example of what I’m talking about. RSS is nice and simple, and as a result has seen widespread industry adoption. Crucially too there are many applications that consume RSS as well as just produce it, which is something not many other formats can boast.

Our view is that while many of these formats are academically brilliant and conceptually very clever, but they are just too complicated.

Ben informed me of this just as I read this comment on an old interview he did on the subject of data portability.

There’s your answer clappingtree, albeit a little late!

So it seems like a good time for me to blog about one of the funky things I’ve been working on recently, the Open Data Definition (ODD).

So, what is ODD?

ODD is an XML based data exchange format which is designed to be simple to implement and use. It consists of a framework and an extension format defining keywords.

The development of ODD fell out of a need for import/export functionality in the new version of Elgg. Import/export was one of the most requested features for previous Elgg incarnations, but be quickly realised that it could be converted into something that had many more uses.

As covered in the Economist article, Data silos don’t cut it anymore. Users want to be able to move their accounts between social networks and have friends on different networks without having thousands of accounts (an issue we looked at solving in a slightly different way with explode).

When looking for a solution we did look at adapting one or more of the existing data portability solutions, but to say that none of them seemed suitable was somewhat of an understatement. Many seemed to fall foul of a problem common… they are just too damn complicated for widespread adoption!

ODD, as mentioned previously, is XML based. When making it I used Elgg 1′s object model as a guide and reduced things down to their lowest common denominators, therefore we have three main components – Entities, Metadata and Relationships.

These components are atomic, and the format itself has virtually no nesting. This is slightly unconventional, but it makes the format easy to parse, supports partial import/export and makes it easy to extend the format to support the live pinging of updates.

This gives us:

Entity

Entities are “things”, for example a web log post or a user account. The entity has a “class” attribute to specify what type of entity it is and can be subclassed.

All entities are identified by a UUID, this is important and I’ll get on to that later.

Metadata

Metadata provides information about an entity as a name/value pair. Optionally, you can give a type to specify the type of metadata – e.g. attribute or annotation.

Relationship

As the name suggests, a relationship defines the relationship between two entities. To do this they use a “verb” (as defined in the extension format mentioned above). Doing it this way permits setting and un-setting operations – for example, friend & unfriend, join & leave.

The UUID

An important concept in all this is the UUID.

The UUID is a URL which must point to an ODD representation of the thing it represents. I think this is quite a powerful concept since it permits truly distributed networks to be build.

An example

To give you an idea of how this might look, here’s an example ODD document.

<odd>
<header version=”1.0” extension=”SN:1.0” generated=”....” />

<entity uuid=”http://foo.com/export/34/” class=”object” subclass=”blog” />

<metadata entity_uuid=”http://foo.com/export/34/” uuid=”http://foo.com/export/34/metadata/45” name=”owner”>http://foo.com/export/24</metadata>

<metadata entity_uuid=”http://foo.com/export/34/” uuid=”http://foo.com/export/34/metadata/49” name=”content”>I saw Sindy at the mall today, she thinks she's all that, but she's not all that... I'm going to cry now and listen to Emo music.</metadata>

<metadata entity_uuid=”http://foo.com/export/34/” uuid=”http://foo.com/export/34/metadata/4” name=”tag”>angst</metadata>

<metadata entity_uuid=”http://foo.com/export/34/” uuid=”http://foo.com/export/34/metadata/5” name=”tag”>emo</metadata>
</odd>

Pretty simple, I hope you’ll agree!

The only extra thing to note is the header element, which simply gives some version information about the framework and extension being used.

I will be giving a brief presentation about this at the Oxford Geek Night on the 22nd and will be answering questions after the event (and again in San-francisco on May 7th), feel free to come along!

In the meantime, have a look at http://www.opendd.net