The real problems comes with third party plugins which have their own database queries … we strongly recommended people didn’t do this, but there is nothing we can do to stop them, and I know that many did.

I was wondering if I can connect elgg to another type of DB maybe oracle or even a nosql DB, how can I achieve this? even if it needs heavy customization. I would really appreciate is if you give me a head start.

also do u think it is better to do it on 1.7 or on 1.8?

Would be interested in hearing people’s experiences though.

http://www.tmdhosting.com/elgg-cloud-hosting.html

Pre 1.5 there was an issue with scaling where plugins and views were concerned however this has been fixed – this may be what a couple of people have had concerns with in the past.

By way of example – Community.elgg.org is an active site with quite a few users. This site is running on vanilla elgg on a very modest server without any special tweaks. It isn’t even running memcache as it ran fast enough without it.

With memcache enabled hits on the database drop dramatically as well, leaving most of the work done by the apache process.

My advice is still to walk before you can run. Get a good site going and then scale up. All scalability problems are solvable ones, and if it is necessary to perform modifications to the code then these can be accomplished when it is economical to do so (cost of development < new hardware for equiv performance gain).

As you scale the database grows in size of course, so one could look at splitting reads and writes (support already built in). There are also sharding techniques to try but this would require some schema modification.

Comments like the one above by C Marcel scares me: “Whether my understanding of Elgg’s architecture is correct or not, the point is that a system’s performance can degrade exponentially as certain quantities grow. In such scenarios, no matter how much hardware you throw at that problem, you won’t fix it”.

Exponents always rules and … exponents always scares.

I was thinking yesterday: OK, I don’t know much about this, but it looks nice. Lets go for it.

But, if Elgg does not not scale? Do you have any model of the scaling (even the simplest would do)? What is your idea behind scaling?

To be more specific, how many servers (dedicated) are needed per user (or per 5% user mentioned above)? And how does it scale? Exponential use? A benchmark is not needed to make such an estimate.

(And, the site we are designing will have millions of users in the next few years (it is OK if you laugh). 2010 it will be 20 000. And, of course I worry, even if the failure probability for the site is large.)

Best regards,
Hakan Olin

PS And I think your Elgg is fantastic. Really …

You are right that no benchmarks have really been done (and I would be interested in hearing your experiences), but to start you off:

Research done by Stan from Brighton Uni indicates that on a typical to high usage site you can expect to have ~5% of users active at any one time (this is typical usage, obviously you’ll get spikes which you should probably be able to handle).

Memory wise during an individual page impression elgg generates some caching and temporary variables – so you can guess at about memory = site object + site query + user object + user query + dataset of page.

CPU wise, all indications that we have done through profiling is that script execution is a negligible portion of the overall page load – on the order of ~0.02 sec depending on machine.

In terms of hardware, our larger installs are still fairly modest in hardware terms (which is a good thing!), and if you were going to splash out on anything I’d splash out on memory (which gives you memcache options).

I have to install Elgg on a site with 50k users and we expect to have 500k in 2 years time. We need to provide an estimation in the amount of hardware required. I know that there are not real benchmark done and it all depends on the use etc… but do you know how much memory is each logged user consuming ? how many pages can elgg serve per second per CPU etc…. any data would be very usefull. I’ve seen that elgg website has nearly 20k users. What kind of hardware are you using to serve it ? how many servers Apache, MySql in cluster etc ….

thanks !