[Opensim-dev] .net serialization of SOGs

[Mikko Pallari]

On Mon, Oct 11, 2010 at 10:40 AM, Toni Alatalo <antont at kyperjokki.fi> wrote:

> On su, 2010-10-10 at 17:06 -0700, diva at metaverseink.com wrote:
> > Adam ping!
> > How's the SOG/SOP refactor coming along? This is coming very close to
> > the need for the refactoring...
>
> Yes, as I think was even written in the refactor plan, a goal there was
> to get Collada save&load support. So that the scene code is first
> refactored to use components, and then a new serialization system made
> that can do many formats. First .net serialization, but then Collada
> too.
>
> All I know is that Adam has put the very first steps to the soprefactor
> branch, a little addition that allows to store arbitrary data in
> components in a map attached to prims. The refactor is not done, the
> hardcoded prim data is still there etc. like before. With the refactor
> the idea is that all data would be in smaller components, for example
> the prim shape data is in an (optional) prim component etc. It seems
> that Adam is so be busy with other things, that if we want the refactor
> implemented someone else should probably step up and do it.
>
> About the formats, yes, something that is not an autogenerated dump of
> the internal code structures that AFAIK every dev wishes that would be
> changed (sog/sop) would be good. A year ago I almost started
> implementing OAR read&write support to Blender, but then when actually
> looked at the "format" decided otherwise and looked at the Collada scene
> spec which seemed quite sane. We've since ended up using Ogre .scene
> files 'cause it was most straightforward for now (dead simple xml scene
> tree, http://www.ogre3d.org/tikiwiki/DotScene).
>
> One suggestion from the talk we had on irc on #opensim-dev now:
> < k\o\w> are you using collada to wrap entire scenes or for individual
> meshes?
> < antont> entire scenes
> < k\o\w> I agree that OAR encapsulation should be replaced with collada
> < k\o\w> but that the contained prim data still needs a saner format
> like HPA
> 09:49 < antont> k\o\w: agreed
>
> We are currently using Collada only when testing WebGL rendering, by
> exporting old scenes from Blender that we've previously exported to
> opensim+modrex using Ogre .scene and .mesh etc, to Collada .dae that
> GLGE can load. But that is giving some more experience with the format
> at least, and so far seems good. Of course it is not suitable out of the
> box for OAR like usage where we need all the opensim specific bits, like
> permissions etc. So perhaps Collada is for import&export, not for backup
> and internal things. Unless OAR would be an extension to it somehow, the
> x is for extensible after all :) . So I guess your plan to do continue
> with the .net xml format but with more control is good.
>
> I'm now about to diverge from your original question and the exact task
> at hand, but perhaps it is helpful to explain a bit what we've done now
> with scene formats and storing data to opensim to know what kind of
> requirements and solutions there are around:
>
> The goal that we have had for which would need the sog/sop refactor,
> allowing storage of arbitrary data in scenes, we've implemented in
> ModRex as a temporary measure so that the client development has been
> able to continue.
>
> The first solution made in March reused an old rex thing that had been
> there forever, RexFreeData, which allowed storing max 1024 chars of text
> attached to an object (to give params to scripts and for them to store
> vals etc). That first impl, used in Naali 0.2 & 0.3 series, just stores
> custom scene entity data as xml in that field and that works also
> against the old pre-naali&taiga realxtend servers 'cause that freedata
> mechanism was there already. But it is limited to that 1024 bytes
> ('cause uses a single generic message for synching all data for an
> entity, and in lludp messages are max 1024 long).
>
> Now, out since last week, there is a new implemention for custom scene
> data storage in modrex. It has tables for entities, components and
> attributes and new messages for doing per-attribute synchronization etc.
> This is out in Taiga 0.2.0 rc1, but not implemented in Naali yet (I
> guess the guys do the client side soon now?). Because it does per-attr
> sync, I think is limited at 1024 for the size of an individual attr, but
> doesn't have a limit for the aggregated size of all attributes of all
> components that are attached to an entity. So should work for pretty
> much everything.
>

Small correction in here. We have new table only for component data, not for
entities or attributes. And we have messages for synchronization of
individual components, not attributes. As said, these have now been
implemented in ModreX only, but will be implemented in Naali soonish. The
size of the component is only limited by the maximum capacity of the udp
generic message.

Cheers,
Mikko


>
> We hope that this kind of functionality becomes supported by Opensim
> core so that also the data of other applications, and not only the data
> specified by the Second Life (tm) application, gets stored in OARs and
> works when rezzing things from inventory etc. And here the application
> does not mean some realXtend impl, but what *you* or any user defines as
> the components & attributes of your application. For example, I have a
> minimal test component 'Door' with two boolean attributes: opened/closed
> & locked/unlocked. That works on the platform, but the platform doesn't
> need to know anything about doors (in fact is not allowed to know, but
> has to be generic and allow any kind of data).
>
> In the spirit of what Melanie wrote about LLUDP independence, we have
> now also an experimental branch (called Tundra) which doesn't depend on
> LLUDP but uses another protocol (kNet, eNet like) for comms and the
> generic Entity-Component attribute sync *exclusively* for all data. For
> storage it uses a generic mechanism that serializes all
> entity-components with their attribute values, either as xml or to a
> binary file or to a db. That's why the EC mechanism has to be used to
> all data, otherwise it doesn't get stored :)
>
> That is now developed against a server module written for Naali itself
> (in that same Tundra branch on github), but the idea is that if the
> protocol and the approach in general proves useful, but the simple
> server misses features that Opensim has and people need, the same
> protocol could then be implemented as a ClientStack + View to Opensim
> too. The experiment is becoming operational now and hopefully works for
> some basics within a month or so, so we'll know more soon'ish.
>
> 2cently yours,
> ~Toni
>
> > On 10/10/2010 4:45 PM, Justin Clark-Casey wrote:
> > > On 09/10/10 19:05, diva at metaverseink.com wrote:
> > >> Dear devs,
> > >>
> > >> I'm pulling my hair here with the serialization of scene objects. This
> > >> is horrible! Using .Net serialization for something as important as
> this
> > >> is the worst idea ever. We're completely frozen. I'm trying to add an
> > >> additional field for the creator info, and I'm stuck in all sorts of
> > >> ways: I can't compute the value of that field at serialization-time,
> and
> > >> then there are the issues of compatibility with previous versions of
> the
> > >> SOP class.
> > >>
> > >> I'd like to understand all the implications of doing an entirely
> > >> different serialization procedure, one that does not use reflection,
> and
> > >> that allows for processing-during-serialization. What will break, and
> > >> therefore needs fixing?
> > >
> > > Grovelling back through the archives, I found the e-mail where I
> > > proposed this very thing back in June 2009!
> > >
> > > https://lists.berlios.de/pipermail/opensim-dev/2009-June/006805.html
> > >
> > > And here's the old patch which partially implemented write.
> > >
> > > https://lists.berlios.de/pipermail/opensim-dev/2009-July/006954.html
> > >
> > > As you wrote later, some equivalent code was later put in libOMV,
> though
> > > I would test it carefully (at the time I vaguely recall thinking that
> it
> > > didn't deal with some serialization cases, but this may well have been
> > > addressed).
> > >
> > > I'm not excited about whether this code remains in libOMV or is placed
> > > in OpenSim if it's going to act as a legacy format in some way.
> > >
> > > As far as I recall, there are no implications of doing this manually as
> > > long as the XML ends up being the same.
> > >
> > >>
> > >> I understand all archiving related to previous versions will break, so
> > >> we need to keep supporting the existing serialization 'disease' for
> the
> > >> foreseeable future. I also understand that TPs/crossings between sims
> > >> that talk different SOG serializations will break, but that's not so
> bad.
> > >
> > > If the SOG serialization is going to change in an incompatible manner,
> I
> > > absolutely need to bump the OAR/IAR version numbers at the same time in
> > > order to maintain confidence in the archiving. I also need to write the
> > > code to stop old OpenSims trying to load these new versions.
> > >
> > > Please can we co-ordinate this in a branch?
> > >
> > >>
> > >> Anything else I should be aware of before I go off and redo this?
> > >
> > > Although .net auto-serialization is fatal if you want to make type
> > > changes to existing fields, it has been very convenient for adding new
> > > fields, since extraneous xml elements are simply ignored on
> > > deserialization. I think it's almost essential that a new format behave
> > > in the same way.
> > >
> > > Also, are you going to put version numbers in the new serializations so
> > > that any future incompatible can be dealt with in some manner?
> > >
> > > Is it worth considering whether other virtual environments might store
> > > completely different object serializations?
> > >
> > > For any new format, I agree with Dahlia in that something OSD-like may
> > > be worth looking at, perhaps just a simple map as I was considering for
> > > the 'dynamic attributes' in SOP. In fact, this could allow that to fit
> > > in nicely.
> > >
> > _______________________________________________
> > Opensim-dev mailing list
> > Opensim-dev at lists.berlios.de
> > https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20101018/253a153c/attachment-0001.html>