[Opensim-dev] switch gears a bit, thoughts on the state of the database in OpenSim

[Sean Dague]

Over the past couple of months I've been working on an nhibernate
provider in between random bug fixes and other clean ups.  For those
that don't know, nhibernate is a generic db mapping layer that helps
smooth out differences between database engines.

There are a few things I found in this process.

First off, it took progressively longer to do the Asset -> Inventory ->
User (never quite finished) -> Region (very partial) stores for
nhibernate.  Those paying attention will note that basically that
corresponds to the reverse chronological order of those
interfaces/objects.  As we go back in time to how our objects were
defined, cruft levels and inconsistencies spring up.  When mapping all
of this out with custom code, these just get code around with more
custom code.  When trying to push through a more standard interface,
these inconsistencies become much more evident and time sinking.

Secondly, a lot of our older objects definitely don't follow best
practices of having properties, and a lot of refactoring was done along
the way.

Thirdly, after tuning some of the mysql tables, the nhibernate bits
definitely seemed a bit slower than they should be.  We are using 1.2
(2.0 is soon to be out).  While this is most likely something we can
actually make better, there is an bit of a hit in the naive case for
doing this approach.

Forthly, nhibernate can't do automated table upgrades (even in 2.0).
The lack of this was the reason for writing the Migrations framework
which seems to be working out quite well across all our data sources.
This was something I was hoping to get for free which we didn't, sadly.
 This also has let us do data optimization (like the fixed length
uuids), and add indexes sanely.

Finally, prevailing feeling in the community right now is that we've got
sqlite for easy startup, and most people are using mysql for serious
work.  I think with moving towards addons we'll actually be able to have
out of tree drop ins for other data sources.

All of this has led me to a slight change of gears.... I'm going to be
putting the nhibernate stuff on the back burner for now and focus on a
few other things that I think will be of more near term use (and will
make completion of nh or other db drivers easier).

 * Some clean up of the scene object model.  We iterate over entities
 way too often in scene, which is an issue from both program correctness
and performance.
 * The rest of the UUID cleanup in the existing db drivers.  We're down
to 2 formats for UUID, but we really need to get to one.  That's a chunk
of time.
 * Normalizing the various data driver interfaces.  It's sort of amazing
how different each of those is in both flavor and naming convention,
which causes substantial confusion.
 * Experiment with the possibility of doing SOP recursively and getting
rid of SOG.  Before anyone freaks out, this will just be an experiment,
and will be presented to the group for review prior to anything that
extensive going into trunk.

I think that a lot of this is pretty critical right now for a couple of
reasons.  Firstly, that doing a new db driver requires a bit too much
work, which is bad for all of us.  Secondly, if we are going to actually
standardize on an XML3 export format (which I think is a really
important thing to do this fall), it would be really good if some of the
base objects we use today we a bit more normalized and encapsulated.

Anyway, I wanted to at least let folks know where my head was at, and
what I'm planning on focusing in the near term (given time as the
biggest limited resource).  If anyone else wants to dig in on what's
left on nhibernate to do, I'd be happy to point you in the right
direction.  I've learned a lot about the way the system works, and most
of the gotchas.  Finishing it off isn't rocket science, it's just time.

	-Sean

-- 
Sean Dague / Neas Bade
sdague at gmail.com
http://dague.net


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20080828/928b503e/attachment-0001.pgp>