[Opensim-dev] .net serialization of SOGs

Mon Oct 11 07:40:40 UTC 2010

On su, 2010-10-10 at 17:06 -0700, diva at metaverseink.com wrote:
> Adam ping!
> How's the SOG/SOP refactor coming along? This is coming very close to 
> the need for the refactoring...

Yes, as I think was even written in the refactor plan, a goal there was
to get Collada save&load support. So that the scene code is first
refactored to use components, and then a new serialization system made
that can do many formats. First .net serialization, but then Collada
too.

All I know is that Adam has put the very first steps to the soprefactor
branch, a little addition that allows to store arbitrary data in
components in a map attached to prims. The refactor is not done, the
hardcoded prim data is still there etc. like before. With the refactor
the idea is that all data would be in smaller components, for example
the prim shape data is in an (optional) prim component etc. It seems
that Adam is so be busy with other things, that if we want the refactor
implemented someone else should probably step up and do it.

About the formats, yes, something that is not an autogenerated dump of
the internal code structures that AFAIK every dev wishes that would be
changed (sog/sop) would be good. A year ago I almost started
implementing OAR read&write support to Blender, but then when actually
looked at the "format" decided otherwise and looked at the Collada scene
spec which seemed quite sane. We've since ended up using Ogre .scene
files 'cause it was most straightforward for now (dead simple xml scene
tree, http://www.ogre3d.org/tikiwiki/DotScene).

One suggestion from the talk we had on irc on #opensim-dev now:
< k\o\w> are you using collada to wrap entire scenes or for individual
meshes?
< antont> entire scenes
< k\o\w> I agree that OAR encapsulation should be replaced with collada
< k\o\w> but that the contained prim data still needs a saner format
like HPA
09:49 < antont> k\o\w: agreed

We are currently using Collada only when testing WebGL rendering, by
exporting old scenes from Blender that we've previously exported to
opensim+modrex using Ogre .scene and .mesh etc, to Collada .dae that
GLGE can load. But that is giving some more experience with the format
at least, and so far seems good. Of course it is not suitable out of the
box for OAR like usage where we need all the opensim specific bits, like
permissions etc. So perhaps Collada is for import&export, not for backup
and internal things. Unless OAR would be an extension to it somehow, the
x is for extensible after all :) . So I guess your plan to do continue
with the .net xml format but with more control is good.

I'm now about to diverge from your original question and the exact task
at hand, but perhaps it is helpful to explain a bit what we've done now
with scene formats and storing data to opensim to know what kind of
requirements and solutions there are around:

The goal that we have had for which would need the sog/sop refactor,
allowing storage of arbitrary data in scenes, we've implemented in
ModRex as a temporary measure so that the client development has been
able to continue. 

The first solution made in March reused an old rex thing that had been
there forever, RexFreeData, which allowed storing max 1024 chars of text
attached to an object (to give params to scripts and for them to store
vals etc). That first impl, used in Naali 0.2 & 0.3 series, just stores
custom scene entity data as xml in that field and that works also
against the old pre-naali&taiga realxtend servers 'cause that freedata
mechanism was there already. But it is limited to that 1024 bytes
('cause uses a single generic message for synching all data for an
entity, and in lludp messages are max 1024 long).

Now, out since last week, there is a new implemention for custom scene
data storage in modrex. It has tables for entities, components and
attributes and new messages for doing per-attribute synchronization etc.
This is out in Taiga 0.2.0 rc1, but not implemented in Naali yet (I
guess the guys do the client side soon now?). Because it does per-attr
sync, I think is limited at 1024 for the size of an individual attr, but
doesn't have a limit for the aggregated size of all attributes of all
components that are attached to an entity. So should work for pretty
much everything.

We hope that this kind of functionality becomes supported by Opensim
core so that also the data of other applications, and not only the data
specified by the Second Life (tm) application, gets stored in OARs and
works when rezzing things from inventory etc. And here the application
does not mean some realXtend impl, but what *you* or any user defines as
the components & attributes of your application. For example, I have a
minimal test component 'Door' with two boolean attributes: opened/closed
& locked/unlocked. That works on the platform, but the platform doesn't
need to know anything about doors (in fact is not allowed to know, but
has to be generic and allow any kind of data).

In the spirit of what Melanie wrote about LLUDP independence, we have
now also an experimental branch (called Tundra) which doesn't depend on
LLUDP but uses another protocol (kNet, eNet like) for comms and the
generic Entity-Component attribute sync *exclusively* for all data. For
storage it uses a generic mechanism that serializes all
entity-components with their attribute values, either as xml or to a
binary file or to a db. That's why the EC mechanism has to be used to
all data, otherwise it doesn't get stored :)

That is now developed against a server module written for Naali itself
(in that same Tundra branch on github), but the idea is that if the
protocol and the approach in general proves useful, but the simple
server misses features that Opensim has and people need, the same
protocol could then be implemented as a ClientStack + View to Opensim
too. The experiment is becoming operational now and hopefully works for
some basics within a month or so, so we'll know more soon'ish.

2cently yours,
~Toni

> On 10/10/2010 4:45 PM, Justin Clark-Casey wrote:
> > On 09/10/10 19:05, diva at metaverseink.com wrote:
> >> Dear devs,
> >>
> >> I'm pulling my hair here with the serialization of scene objects. This
> >> is horrible! Using .Net serialization for something as important as this
> >> is the worst idea ever. We're completely frozen. I'm trying to add an
> >> additional field for the creator info, and I'm stuck in all sorts of
> >> ways: I can't compute the value of that field at serialization-time, and
> >> then there are the issues of compatibility with previous versions of the
> >> SOP class.
> >>
> >> I'd like to understand all the implications of doing an entirely
> >> different serialization procedure, one that does not use reflection, and
> >> that allows for processing-during-serialization. What will break, and
> >> therefore needs fixing?
> >
> > Grovelling back through the archives, I found the e-mail where I
> > proposed this very thing back in June 2009!
> >
> > https://lists.berlios.de/pipermail/opensim-dev/2009-June/006805.html
> >
> > And here's the old patch which partially implemented write.
> >
> > https://lists.berlios.de/pipermail/opensim-dev/2009-July/006954.html
> >
> > As you wrote later, some equivalent code was later put in libOMV, though
> > I would test it carefully (at the time I vaguely recall thinking that it
> > didn't deal with some serialization cases, but this may well have been
> > addressed).
> >
> > I'm not excited about whether this code remains in libOMV or is placed
> > in OpenSim if it's going to act as a legacy format in some way.
> >
> > As far as I recall, there are no implications of doing this manually as
> > long as the XML ends up being the same.
> >
> >>
> >> I understand all archiving related to previous versions will break, so
> >> we need to keep supporting the existing serialization 'disease' for the
> >> foreseeable future. I also understand that TPs/crossings between sims
> >> that talk different SOG serializations will break, but that's not so bad.
> >
> > If the SOG serialization is going to change in an incompatible manner, I
> > absolutely need to bump the OAR/IAR version numbers at the same time in
> > order to maintain confidence in the archiving. I also need to write the
> > code to stop old OpenSims trying to load these new versions.
> >
> > Please can we co-ordinate this in a branch?
> >
> >>
> >> Anything else I should be aware of before I go off and redo this?
> >
> > Although .net auto-serialization is fatal if you want to make type
> > changes to existing fields, it has been very convenient for adding new
> > fields, since extraneous xml elements are simply ignored on
> > deserialization. I think it's almost essential that a new format behave
> > in the same way.
> >
> > Also, are you going to put version numbers in the new serializations so
> > that any future incompatible can be dealt with in some manner?
> >
> > Is it worth considering whether other virtual environments might store
> > completely different object serializations?
> >
> > For any new format, I agree with Dahlia in that something OSD-like may
> > be worth looking at, perhaps just a simple map as I was considering for
> > the 'dynamic attributes' in SOP. In fact, this could allow that to fit
> > in nicely.
> >
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev