Date

Jan. 2, 2013

Status (draft, in progress, done or abandoned)

Done.

Proposers

Originally implemented by JustinCC in 2010.

Being revived by Oren Hurvitz and JustinCC.

Implemented since OpenSimulator 0.7.6.

Introduction

This is a proposal to add a way of attaching arbitrary data to any Scene Object. The feature has actually already been implemented by JustinCC, but it was never pushed to the master branch. I would like to move this feature into master, and fill in any missing functionality (e.g., serialization).

Proposal

Original January 2013 proposal revival

(more recent discussion is at http://lists.berlios.de/pipermail/opensim-dev/2013-January/011624.html)

Here's Justin's original message about this feature (from http://lists.berlios.de/pipermail/opensim-dev/2010-August/009166.html):

Hi folks. As I mentioned last week, I have now pushed a dynamic-attributes branch to the Git repository. This is work that I was experimenting with while doing media-on-a-prim.

Essentially, this allows region modules to store and retrieve data on SceneObjectParts and PrimitiveBaseShapes without adding new attributes to those core classes.

Both SOP and PBS have an OSDMap property (which for various reasons is wrappered by OpenSim.Framework.DAMap). Modules can insert any class derived from libopenmetaverse's OSD class in this map (this includes OSDMap, OSDString, OSDInteger, etc. See the StructuredData.cs file in libopenmetavere's OpenMetaverse.StructuredData/ directory for more details.

Persistence is achieved by [de]serializing this map to xml and storing it in a DynAttrs column in the database. Persistence is implemented for sqlite, mysql and mssql.

There is a small region module in OpenSim.Region.CoreModules/Framework/DynamicAttributes/DAExampleModule.cs that demonstrates this. Every time an object is moved in the scene, it adds 1 to SOP.DynamicAttributes["moves"] and tells all users in the region how many times the object has moved. If the server is restarted, the number of moves is restored on restart.

It didn't take much code to achieve this. However, as discussed previously, there is some awkardness in using OSD in this way. Any feedback on this or other aspects of dynamic attributes is appreciated.

This branch is a 'technology preview' at most and should not be relied upon in production in any way whatsoever.

Recent March 2013 Considerations

Oren's propsal to store dynamic attributes as a JSON serialization rather than XML have triggered me (justincc) to do a thorough review of my original dynamic attributes work and outline the various serialization alternatives. These are

Serialize OSD as JSON

This would serialize the dynamic attributes OSD to a JSON serialization which would then be stored in the database and be embedded as JSON in the XML serialization.

Pros

• Considerably less verbose than the LLSD XML serialization.
• Using OSD makes it somewhat easier to strongly hint that not much at all should be stored in dynamic attributes.
• Easily embedded from other sources since, unlike LLSD XML, it uses native JSON structures (object, number, array, etc.)
• Can be accessed directly by scripts using JsonStore.
• No extra work required.

Cons

• Embedding JSON inside XML makes it harder to human read that serialization when necessary (e.g. the embedded JSON will not be properly indented).
• Embedding JSON inside XML makes it harder for other systems to consume that serialization (need two parsers, separate parsed data structures, etc.)
• Any property change in OSD requires creation of new classes (OSDInteger is immutable, for instance). If we went right down the JSON route, however, we could conceivably dispose of OSD and serialize directly (and hence use native value types).

Serialize OSD as LLSD

This would serialize the dynamic attributes OSD as an LLSD representation (which is XML). This is what we are doing already.

Pros

• Embedding XML in the scene object XML makes it easier to human-read the structure.
• Embedding XML in the scene object XML makes it easier for external code to consume and manipulate.
• Using OSD makes it somewhat easier to strongly hint that not much at all should be stored in dynamic attributes.
• Possible to later change to a JSON serialization for the entire object without compatibility issues/extra module work (under restrictions as noted below).
• Can be accessed directly by scripts using JsonStore.
• No extra work required.

Cons

• More verbose than the JSON serialization, though this would be ameliorated by compression (which we really should be doing everywhere).
• Any property change in OSD requires creation of new classes (OSDInteger is immutable, for instance).

Notes

• For any later JSON representation, need to enforce that non JSON structures like OSDDate are not used.

Serialize OSD as a more concise XML representation

This would still serialize the dynamic attributes OSD as XML but more concisely than is done by LLSD. Perhaps something like

<ossd>
  <map name="MyStore">
    <int name="the answer">42</int>
  </map>
</ossd>

Pros

• Using OSD makes it somewhat easier to strongly hint that not much at all should be stored in dynamic attributes.
• Possible to later change to a JSON serialization for the entire object without compatibility issues/extra module work (under restrictions as noted below).
• Can be accessed directly by scripts using JsonStore.

Cons

• Still more verbose than the JSON serialization, though this would be ameliorated by compression (which we really should be doing everywhere).
• Any property change in OSD requires creation of new classes (OSDInteger is immutable, for instance).
• Requires considerable extra work, and possibly changes to libomv.

Notes

• For any later JSON representation, need to enforce that non JSON structures like OSDDate are not used.

Serialize dynamic attributes as custom XML supplied by the module

This option would see us do away with OSD entirely in favour of allowing modules to read and write plain old XML. For example

<MyStore><the-answer>42</the-answer></MyStore>

Pros

• Less verbose than the highly structured XML representations above.
• Allows direct embedding of XML from other sources.
• Allows direct external code reuse from other sources for XML parsing.
• Allows use of all XML features (e.g. mixed content, namespaces).
• Removes the complexity and cost of manipulating an OSD structure.

Cons

• Makes it impossible to transparently allow a future JSON scene object representation.
• Cannot be directly accessed by scripts using JsonStore without extra work.

Discussion

On thinking about this (justincc), I was initially attracted to storing XML directly rather than forcing it to be within an OSD structure. We would no longer force data to be in the obscure LLSD format and we would also not have the issues (imo) surrounding embedding JSON in XML. It would also allow XML formats from other sources to be directly reused.

However, it also makes it impossible to easily provide a JSON serialization at some future point (how realistic this idea is I don't know). But I think we also need to bear in mind that a dynamic attributes storage mechanism cannot currently replace storing larger chunks of data as assets (I think there may actually be some very interesting ideas around compressing multiple files at once and eliminating this issue but I don't think that can be done at the moment).

Therefore, I think that the prospect of directly reusing other XML formats and serialization size becomes less important. Also, custom XML starts becoming a little tricky to size limit - you would have to always serialize to find out the size (imo, I think with the fixed data types in OSD there is some possibility of doing this without having to serialize).

I'm still highly reluctant to see JSON embedded in XML, and producing a slightly smaller XML serialization of OSD than LLSD isn't worth the costs involved in my opinion. So I think LLSD remains the best highly imperfect choice.

In the future, I think there is scope for serializing a scene object entirely to JSON instead of XML, for receivers that support this. Going with LLSD preserves this choice.

Other issues

Locking

The concurrency story for dynamic attributes is currently very weak. The entire DAMap is locked on writing the serialization and any module should do the same in order to avoid race conditions. Even the DAExampleModule does not get this right as it manipulates it's section of the OSD directly without locking.

This probably should be improved by just locking the individual OSD sections on write.

Serialization

1. The naive approach at the moment is to reserialize the entire OSD structure any time that backup on an object runs. This may not be an issue if one assumes that the amount of data will be low.

This could be improved if we somehow detect whether the OSD has actually been updated.

2. Currently, all data is seralized to a single field (DynAttrs) in the prims table, which means that any change whatsoever requires the entire DAMap to be reserialized. This could be improved by serializing separate OSD module namespaces to a separate region database table (perhaps DynAttrs) with identifying fields (prim ID and key). Whether this is an issue depends on the amount of data, bearing in mind that only a minimal data set should be stored in dynamic attributes.

Storing dynamic objects in the SceneObjectPart rather than manipulating OSD directly

A big issue with DAMap at the moment is that you can only store OSD structures. It might be much better if modules can store arbitrary objects instead, which would be vastly more flexible. They would then register on a serialization event and provide the serialization on demand using the objects in their section of the dynamic attributes map.

Actions

• Improve locking.
• Consider serialization issues.
• Look at storing objects in dynamic attributes rather than an OSD.

Now starting to explore these issues in git master. See http://opensimulator.org/mantis/view.php?id=6541 for more details.