[Opensim-dev] Standardizing OpenSim data formats

[Hurliman, John]

Thank you for taking a look. Looks like I all of the "ExtraData" for prims, including sculpt/particles/flexi/etc. I'll see what OpenSim is doing for saving script state and add everything in a second pass. I'm hoping that this particular format captures everything needed to save simulator state to disk for OpenSim and load it back up, while being easy to create and parse by third party tools. Additional formats could be made later for more specific tasks such as saving assets offline (where things like InventorySerial definitely wouldn't matter).

There could be a couple of different approaches to distinguishing between different content types. OAR loading is already doing this for terrain with a switch statement on the file extension. That works until two separate formats both decide to use the .prim or .xml extensions. Another possibility would be to require that all object serializations use the same format so a MIME type could be read from metadata embedded in the file. Using the OAR index file to store a list of filenames and content types could also work. This breaks some of the nice things about OAR files though, such as opening one up in a file viewer and dropping some content in without using special tools.

John

>-----Original Message-----
>From: opensim-dev-bounces at lists.berlios.de [mailto:opensim-dev-
>bounces at lists.berlios.de] On Behalf Of Justin Clark-Casey
>Sent: Wednesday, April 29, 2009 10:44 AM
>To: opensim-dev at lists.berlios.de
>Subject: Re: [Opensim-dev] Standardizing OpenSim data formats
>
>Hurliman, John wrote:
>> Currently, OpenSim server<->server communication is primarily made up
>of .NET XML serialization. As a side effect of this, I've seen the wire
>format between simulator and grid services or grid service to grid
>service change many times since I started working on backend services.
>The OAR file format has also changed at least once while retaining the
>same version number because the underlying prim XML format changed. I am
>working on a libomv OAR reader/writer at the moment, and trying to deal
>with hacks for supporting pre and post change OAR files.
>>
>
>OAR was intended more as a packaging mechanism than a complete format.
>This is partly why I didn't bump the version
>number when the serialization format changed.
>
>In principle, I think it would be good if the serializations themselves
>identified themselves as one format or another
>(as I think you've already indicated in a previous post).  Then one
>could have different serializations (e.g. mesh)
>without having to produce different OAR structures or different formats
>entirely.  I'm not sure how pragmatic this is -
>perhaps some indication of the format of serialization could be placed
>in the OAR control file (though even then there
>might be regions containing multiple different serializations).
>
>The XML serialization 'XML2' format was used for objects in the OAR
>partly just because it was already there and it
>worked, even if it is pretty horrible.  I feel that one nasty point
>about it for offline content transfer is that it
>doesn't contain any versioning information to identify it (necessitating
>heuristics I guess).  I feel that there may
>also come a point where an offline format will diverge from an on-the-
>wire format (e.g. inventory serial seems
>unncessary for offline purposes - though many other things will remain
>for the backup use case as opposed to the
>transfer to some one else use case).
>
>> To increase compatibility between versions of OpenSim and with third
>party services I'd like to propose a standardization of (eventually) all
>over-the-wire communication in OpenSim. I started with an attempt at an
>LLIDL definition for primitives:
>>
>> http://opensimulator.org/wiki/PrimitiveFormatProposal
>>
>> Is anything missing? Does this look like a reasonable start?
>
>On a quick look through, I don't see any attribute for sculpt
>information.  At the moment this is stored inline in the
>serialization.  Also, I suspect state information for scripts may have
>crept in recently (not sure).
>
>Also, at least for archiving purposes, the creator id will at some point
>become a string rather than a uuid in order to
>allow identification information to be retrieved on grids other than the
>one on which the archive was saved.
>
>--
>justincc
>Justin Clark-Casey
>http://justincc.wordpress.com
>_______________________________________________
>Opensim-dev mailing list
>Opensim-dev at lists.berlios.de
>https://lists.berlios.de/mailman/listinfo/opensim-dev
>