[Opensim-dev] JSON or XML for serialization in the OpenSim database?

Tue Jul 6 19:46:44 UTC 2010

I believe performance is important here. I also believe that it is
more important even that human readability, as anyone who has a need
to read it also has the skill to build a parser. I think
human-readability is overrated, it benefits developers, but not users.

Melanie

Hurliman, John wrote:
> I think the original question has been misunderstood. We are using the LLSD type system through the OpenMetaverse.StructuredData.dll library. If we want to decide whether that is a good approach or not we can open the discussion, but the question originally posed in this thread was a lot smaller scope. Under the assumption that we are using LLSD (primarily the OSDMap class), which serialization format should be used? There are three possibilities: XML, JSON, and binary. Since we're dealing with an LLSD representation on the OpenSim side, there are no DTDs here. There is no XPath/XQuery, no Javascript evaluation, no "conversion skipping" by saving raw data from clients straight to the database and sending it back without going to an intermediary representation in memory (it's arguable whether that is a good idea at all, but a moot point since it's not how our current type system works).
> 
> re worthy of consideration are human-readability (debugging and digging through MySQL rows are par for the course here) and performance (both in storage/bandwidth cost and parsing/serialization tax). If the goal is to switch to pure XML and use DTDs instead of LLSD with LLIDL or an equivalent form of documentation then let's put that on the table instead of comparing apples to oranges.
> 
> My vote for the original question goes to JSON, as a good balance between performance and readability. I think the performance impact is being underestimated. I have a test implementation of linkset serialization using LLSD/JSON that shrinks the on the disk representation of ~50KB objects down to a few KB. The parsing time also goes down in a similar fashion. The real-world (virtual world?) impact here means shaving hundreds of milliseconds off border crossings, making cross-grid functionality like HyperGrid work better, and making serializations like OAR/IAR more usable. The performance gap is not something to hand-wave away.
> 
> John
> 
> -----Original Message-----
> From: opensim-dev-bounces at lists.berlios.de [mailto:opensim-dev-bounces at lists.berlios.de] On Behalf Of Frisby, Adam
> Sent: Monday, July 05, 2010 5:14 PM
> To: opensim-dev at lists.berlios.de
> Subject: Re: [Opensim-dev] JSON or XML for serialization in the OpenSim database?
> 
> Having just worked on a JSON project myself internally - I personally developed a bit of a loathing for the format. I'm personally partial to XML, ideally with a corresponding DTD.
> 
> Adam
> 
>> -----Original Message-----
>> From: opensim-dev-bounces at lists.berlios.de [mailto:opensim-dev- 
>> bounces at lists.berlios.de] On Behalf Of Teravus Ovares
>> Sent: Monday, 5 July 2010 4:23 PM
>> To: opensim-dev at lists.berlios.de
>> Subject: Re: [Opensim-dev] JSON or XML for serialization in the 
>> OpenSim database?
>> 
>> Not a whole lot of feedback here yet, maybe people are on a long 
>> weekend type camping vacation..
>> 
>> I'm partial to OSD/json, myself.    I'd also like to, at some point,
>> get a version number in there along with a definition of the format
>> for people who want to write integration tools..    however, that last
>> bit may be more of a 1.0 thing.
>> 
>> I think a lot of tools are going to go the way of JavaScript in the
>> future for various reasons...   one being that..   it's generally
>> implemented in all web enabled devices.   Computers, 'tablets', 'smart
>> phones'...   Another reason is it's more compact, while still being
>> fairly human readable.     One last reason that I can think of at this
>> moment is there are no external dependencies that 'get lost and turn
>> into a 404', like with XML Schemas.   I've done several XML based
>> integrations using REST and noted that in 55% of the cases, the 
>> defining schema is a 404 which makes validation and automatic creation 
>> of XML Serialization classes impossible.  Worse, in 15% of the cases, 
>> extensions are defined in the schema and then used in the XML..  Only, 
>> you won't ever know what tags and parameters the extensions provide 
>> because the schema is 'missing'.
>> 
>> Regards
>> 
>> Teravus
>> 
>> On Sun, Jul 4, 2010 at 8:28 PM, Justin Clark-Casey 
>> <jjustincc at gmail.com> wrote:
>> > Hi folks,
>> >
>> > As part of the media-on-a-prim implementation, I'm serializing the 
>> > parameters for a media texture to the database.  This seems better
>> than
>> > creating new database fields or even a whole new table for these
>> parameters,
>> > both because there are lots of them (url, scaling, controls,
>> whitelist,
>> > etc.) and because different future virtual environments may want to
>> store
>> > different things.
>> >
>> > I'm going to serialize them as an OSDArray or MediaEntrys using the 
>> > libopenmetaverse library.  However, the question then becomes 
>> > whether
>> to use
>> > the JSON representation or the XML representation.
>> >
>> > I tend to prefer XML for storage representations.  I believe that
>> it's
>> > somewhat more human readable and that there is better tool support
>> for
>> > manipulating it.  However, I know other people would prefer storage
>> in JSON
>> > and I accept that serialization/deserialization there may be 
>> > slightly faster.
>> >
>> > The only other example of serialization that I know of in OpenSim
>> currently
>> > is that of SceneObjectGroups into inventory, which encompasses 
>> > object properties, object inventory properties and script state.  
>> > This is
>> done in
>> > XML and media entries would become part of that serialization.
>> >
>> > If there's a majority preference for JSON I don't mind using that
>> instead,
>> > though I would want a justification for going this route rather than
>> XML.
>> >  If there's no real argument then I will go with XML.
>> >
>> > Also, I believe that we should try and be consistent, so picking one
>> or the
>> > other now should make it more likely that the same approach would be
>> used
>> > for the next serialization case.
>> >
>> > Regards,
>> >
>> > --
>> > Justin Clark-Casey (justincc)
>> > http://justincc.org
>> > http://twitter.com/justincc
>> > _______________________________________________
>> > Opensim-dev mailing list
>> > Opensim-dev at lists.berlios.de
>> > https://lists.berlios.de/mailman/listinfo/opensim-dev
>> >
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
> 
> 