[Opensim-dev] Proposal: Introduce key:value pair dictionaries into SOP and PrimitiveBaseShape
Justin Clark-Casey
jjustincc at googlemail.com
Wed Aug 4 20:38:34 UTC 2010
On 04/08/10 18:23, Hurliman, John wrote:
> Does it actually need to hold arbitrary data types, or would the LLSD type system
> encompass everything you need? It natively handles all the basic C# value types, the
> domain-specific types in OpenMetaverseTypes.dll (vectors/quaternions), plus arrays and maps.
> The main benefit you get from using an OSDMap is transparent serialization of arbitrary data.
> There's no way to serialize Dictionary<string, object> transparently* (aside from digging
> ourselves back into the .NET serialization hole) so you end up with something like
> Dictionary<string, ISerializableData> and define methods on the ISerializableData interface for
> storing and loading an OSDMap anyways. Restricting the type system of data that can be attached to
> SOG/SOPs also makes memory handling and cleanup more straightforward, since you know there are no
> unmanaged pointers or special resources such as sockets attached directly to prims that need cleanup
> before prims are thrown away. Finally, it eases dependency issues since you never have to worry about
> defining a MyExtendedPrimData class in one assembly and trying to typecast to it in another assembly.
> With over 90 different assemblies in OpenSim right now this can cause major headaches.
Yes, I started down the path of trying to serialize an IDictionary<string, object> before realizing that this was going
to be impossible without either a) referencing the classes directly in the core [hence defeating the whole purpose] or
b) getting modules to register for keys in the dictionary and invoking them to handle [de]serialization to specific classes.
Option b) starts getting complicated and I'm not sure how well it would work. It could also incur the other costs that
you mention.
So I started implementing [de]serialization of an OSDMap (Attrs) to hold dynamic SOP attributes. In doing this, I have
the following observations/questions.
1) ## Dynamic attributes are best manipulated directly in their OSD form ##. While one could [de]serialize the OSD
into a native class, this incurs both performance penalties and/or makes the code more complicated. It seems easier in
the end just to manipulate OSD directly.
For instance, under this scheme, instead of taking SOP.Attrs["Media"][<face>] and deserializing it to a libomv
MediaEntry class, I would instead get/set variables as something like
SOP.Attrs["Media"][<face>]["current_url"] = new OSDString("http://example.com");
2) ## Changing OSD attributes is awkward ##. From the example above, you can see that whenever I want to change the
dynamic attribute I need to wrap it in a new OSD class since the code doesn't allow me to change the underlying value.
This is clumsy.
3) ## Stability of OSD attribute names ##. The current_url name above originates from the OSD serialization used for
CAP communication with the client. Could this and other names change? I suspect that any module using OSD would really
have to translate data from the client into its own OSD data structure (even if this was initially identical) in order
to guard against protocol changes.
4) ## OSD/LLSD translates some datatypes (e.g. uint) into byte[] ##. OSD, at least in its LLSD representation, renders
some attributes such as uint into byte[], which harms human readability. I presume this is because it's oriented
towards data transmit rather than storage? This is not ideal for a storage format, at least if one wants some level of
human readability. Perhaps there could be another serialization format that doesn't do these transforms (e.g. stores
uint directly)?
5) ## OSD/LLSD translates some native values into the same datatype ##. For instance, uint, long, ulong and byte[] are
all translated into OSDBinary. This isn't a fatal problem, but it does mean that whoever is using the dynamic attribute
needs to know which native datatype to translate them back into, e.g. OSDBinary.AsLong(), OSDBinary.AsUInteger(), etc.
This is clumsy.
6) ## Attrs locking ##. As OSDMap is a dictionary it would need to be locked on enumeration and hence on update (and
really on read as well on some readings of the Microsoft API docs). This could be awkward unless the OSDMap is wrapped
in a class to lock on access (I already had to wrap it in a class anyway to stop it being auto-serialized).
Even with all these quibbles, I did get as far as implementing dynamic attribute storage for MediaURL and this appeared
to be working fine. However, using the OSD representation of MediaEntry got complicated and so I would rather implement
dynamic attributes separately to reduce complexity. Hence, I've gone back to a conventional implementation of MOAP
SOP/PrimitiveBaseShape attributes.
Sorry for the tl;dr but I think that this is a very interesting area. If we could also save the dynamic attributes to
persistent storage (rather than breaking them up into separate columns) and have a script infrastructure where functions
can be dynamically defined then we could be some of the way to properly separating SL modules and core. Core could be
separated out as a content neutral component rather than as part of an ever-expanding monolith.
Justin
>
> John
>
>> -----Original Message-----
>> From: opensim-dev-bounces at lists.berlios.de [mailto:opensim-dev-
>> bounces at lists.berlios.de] On Behalf Of Justin Clark-Casey
>> Sent: Thursday, July 29, 2010 8:55 AM
>> To: opensim-dev at lists.berlios.de
>> Subject: Re: [Opensim-dev] Proposal: Introduce key:value pair dictionaries
>> into SOP and PrimitiveBaseShape
>>
>> On 29/07/10 01:40, Dahlia Trimble wrote:
>>> Similar to the OSDMap in libomv?
>>
>> Not dissimilar, though the in-memory maps in OpenSim would need to hold
>> arbitrary data types. We couldn't use OSDMap directly unless we were
>> happy to serialize objects to and from OSD for every get and set (or change
>> the values directly in the OSD representation, which might be an idea).
>>
>>>
>>>
>>> On Wed, Jul 28, 2010 at 2:39 PM, Justin Clark-Casey
>>> <jjustincc at googlemail.com<mailto:jjustincc at googlemail.com>> wrote:
>>>
>>> Hi there. Whilst implementing media-on-a-prim, I've been keeping as
>>> much code in the MOAP region module as possible.
>>>
>>> I'm quite impressed with how feasible this is. However, there
>>> remain three major structures where the core of OpenSim has to
>>> understand something about media on a prim.
>>>
>>> 1) Database plugins - to get/put values to named database columns
>>> (e.g. prims.MediaURL).
>>> 2) Script functions (e.g. llGetPrimMediaParams()).
>>> 3) Scene objects (PrimitiveBaseShape.Media and
>>> SceneObjectPart.MediaURL).
>>>
>>> It's difficult to do anything right now about (1) and (2), but I
>>> believe there is an opportunity to address (3).
>>>
>>> What I would like to do is introduce dictionaries into
>>> PrimitiveBaseShape and SceneObjectPart that would supplement
>>> existing fields by storing arbitrary key/value pairs. So instead of
>>> having to hardcode a new MediaURL property on SceneObjectPart I
>>> could instead get/put the data as something like
>>> SceneObjectPart.Values["MediaURL"].
>>>
>>> Thus, the dictionaries can act as blackboards for communication
>>> between plugins and modules without the core of OpenSim having to
>>> get involved. I think that this would move us a tiny way towards
>>> our vision of being a generic virtual environment platform and away
>>> from hardcoded Second Life specifics, and make it easier to write
>>> more ambitious region modules without additions to core.
>>>
>>> Thoughts?
>>>
>>> --
>>> Justin Clark-Casey (justincc)
>>> http://justincc.org
>>> http://twitter.com/justincc
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at lists.berlios.de<mailto:Opensim-dev at lists.berlios.de>
>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>>
>> --
>> Justin Clark-Casey (justincc)
>> http://justincc.org
>> http://twitter.com/justincc
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
--
Justin Clark-Casey (justincc)
http://justincc.org
http://twitter.com/justincc
More information about the Opensim-dev
mailing list