[Opensim-dev] Update for group module & flotsam
Michelle Argus
argus at archimuh.de
Sun Nov 18 18:50:21 UTC 2012
Darn, why did your responce land in my junkmails...^^
The exact same behaviour can be seen in the original and new solution.
In the original solution the time period is just much longer during
which the lag is noticable. The viewer lags the moment the IM is sent
by the viewer and stops once all IMs have been sent out by the
simulator, the issue already starts prior to the simulator actualy
sending the IMs.
During this time the sim console responce is also slowed down when
typing a new command.
I could imagine that it is thread related. In that case tests within
the same datacenter or near by would most probably not cause much
noticable issues. In my Case the distance EU/USA could increase the lag
situation I experiance in a similar way all other requests were we have
increasing issues with slow requests.
Short sidenote on the slow request. The sims are hosted in one of the
top 5 datacenters in germany that we had tested with with very good ping
times to the osgrid servers, so slow requests are unlikely to be caused
by slow network on the simulator side.
Am 10.11.2012 04:44, schrieb Justin Clark-Casey:
> What is the nature of the lag in this case? Is is some freezing of
> the scene (if inbound packets stop being processed) or something
> else? On a naive reading of the code, scene freezing should not
> happen since the IM packet is handled on its own thread and not the
> inbound packet loop thread, though I guess it's possible that for big
> groups, sending a lot of messages may tie up all the available threads
> (not something I have experienced or tested).
>
> Do you see the same thing with the original solution of caching
> presence information directly in the groups service? I would have
> thought so since the sending step in that case is still performed by
> the simulator.
>
> On 08/11/12 22:28, Michelle Argus wrote:
>> NP Justin.
>>
>> I tested the new MessageOnlineUsersOnly and it is a big improvment
>> eventhough some lag still was noticable. The lag is
>> noticable from the moment the IM is beeing sent from the viewer
>> untill all IMs were sent to the online group members.
>> With the new option we have 3 lag creating steps:
>> - Query the groupmembers from the group server ( time is not logged)
>> - Query the presence server for online members (time is not logged)
>> - Sending the IMs (ranged between 59 - 3200 ms for constant 13 online
>> members of 560 total)
>> Same test with false took about 20 seconds to send all IMs
>>
>> As my server is in europe, the 2 queries to the OSGrid group server
>> AND the presence server can both be slow at times.
>> This could be improved if the group service queries the presence
>> server (as in most cases both will be within the same
>> network) instead of the simulator sending both queries. So instead of
>> the currently implemented proposal B, the
>> proposal C would be better for simulators further away or those with
>> slow internet connection.
>>
>> In addition, proposal C does not require each groupmember to be
>> checked for online status if the group server uses a
>> presence cache. Example, if 2 people are sending IMs to diffrent
>> groups with common members, then 1 presence check would
>> be enough for the common members, the lists of agents sent to the
>> presence service would thus also be reduced. As in a
>> bigger grids many IMs are beeing sent gridwide, the presence querries
>> should be even more efficiant in C compared to B
>>
>>
>>
>> Yes, E is a hybrid of B/C and D. It could use the exact same online
>> presence method as in B/C to get the list of online
>> members. The group service would thus not realy need any changes.
>> This also means, that a 3rd party dev could develope
>> the IM relay service which would then not need to be part of the core
>> itself. The only change required would be
>> simulator side, but even this would be a minor change. One would need
>> to implement a url/enable ini setting, and
>> depending on the setting the IM is sent simulator side or passed on
>> to the IM relay service. Its a small change
>> simulator side which should not be a problem for implementation in core.
>>
>> As the IM relay server is an option, everyone can deside themselves
>> if they trust the hoster of the IM relay service or
>> if they want to have their own simulator send all IMs. Talking for
>> the german cummunity in OSGrid, I know that we
>> definatly will use a IM relay to improve the IM situation for the
>> simulator side nomatter if A, B or C get implemented,
>> especialy on our event regions.
>>
>>
>> My personal favorit would be E, were C is the core group service
>> and the optional IM+group notice Service can be run
>> by closed grid admins or 3rd party in open grids to improve worldwide
>> groupchat and notice sending usage.
>>
>>
>>
>>
>>
>> Am 08.11.2012 04:39, schrieb Justin Clark-Casey:
>>> Hi Michelle. Sorry that it's taken quite a long time for me to
>>> reply to this - unfortunately been hit by other work
>>> and divers alarums.
>>>
>>> I somewhat edited [1] again, mainly for sense changes though I also
>>> left a few comments in italics. To try and make
>>> things clearer, I gave the alternatives names (e.g. alternative E
>>> I've called "Separate group IM relay service".
>>> Please change if these are not accurate.
>>>
>>> It would seem that E is mostly a hybrid solution of B (simulator
>>> queries presence service on IM) and D (Groups service
>>> distributes IM)? I would regard this as over-complicated for a core
>>> OpenSimulator solution, especially when one
>>> starts talking about trusted relay services.
>>>
>>> I'm curious if you have tried the experimental
>>> MessageOnlineUsersOnly = true setting I added a couple of weeks ago.
>>>
>>> [1]
>>> http://opensimulator.org/wiki/Feature_Proposals/Improve_Groups_Service
>>>
>>> On 29/10/12 00:13, Michelle Argus wrote:
>>>> I have updated the proposal page and listed the diffrent
>>>> alternative from A onwards sothat its easier for us.
>>>>
>>>> My current favorite is the alternative E: The group server requests
>>>> the online status from the grid server itself and
>>>> caches this data instead of the grid server keeping the group
>>>> server updated.
>>>>
>>>> - Simulators request their data directly from the group server and
>>>> sends IMs itself OR
>>>> - Optionaly the Simulator communicates via a relay service with its
>>>> own cache. The relay service requests its data from
>>>> the same central group server. The relay service can additionaly
>>>> send IMs if wanted to reduce resource usage simulator
>>>> side. The relay service can be hosted by anyone for a worldwide
>>>> network.
>>>>
>>>> The same concept could be used for other services such as assets,
>>>> presence, inventory, friendslist etc which are
>>>> meanwhile causing many issues due to slow requests in bigger grids
>>>> such as OSGrid.
>>>>
>>>>
>>>> Am 23.10.2012 04:40, schrieb Justin Clark-Casey:
>>>>> Apologies, it's [1]. Please feel free to edit it as you see fit
>>>>> - I've put you as one of the proposers. This page
>>>>> is to keep track of the issue rather than a formal proposal
>>>>> mechanism.
>>>>>
>>>>> No rush on this - please feel free to take your time in
>>>>> responding. In truth, I only have a certain amount of time for
>>>>> these issues currently myself.
>>>>>
>>>>> Having messages route through a service rather than be largely
>>>>> handled by simulators themselves is an interesting
>>>>> approach. It's the argument of a distributed versus a more
>>>>> centralized architecture. Although I can't see
>>>>> OpenSimulator going down this route in the near future, if anybody
>>>>> wants to experiment and needs additional config
>>>>> settings then patches are very welcome.
>>>>>
>>>>> [1]
>>>>> http://opensimulator.org/wiki/Feature_Proposals/Improve_Groups_Service
>>>>>
>>>>>
>>>>> On 20/10/12 11:06, Michelle Argus wrote:
>>>>>> Justin, could you post the url to the suggestion page, I think
>>>>>> you forgot to add it ;)
>>>>>>
>>>>>> One issue that having the sim updating online status is, that
>>>>>> if someone has the group module diabled or uses a
>>>>>> diffrent setting then the status is not updated. As other modules
>>>>>> hosted by the grids also might this information, one
>>>>>> should consider adding something to the gridserver for this.
>>>>>>
>>>>>> I also like the idea from Akira to have the groupserver to
>>>>>> receive the full IM and then sending it to everyone instead
>>>>>> of having the sim send the message. One then could have a
>>>>>> specialized server installed for the group module which
>>>>>> cannot
>>>>>> create any lagissues simside. This could then also be used for a
>>>>>> gridwide spamfilter or filtering illigal activities
>>>>>> within the grid.
>>>>>>
>>>>>> Havnt had much time though as I have a longer event running which
>>>>>> ends on sunday...
>>>>>>
>>>>>>
>>>>>> Am 20.10.2012 04:32, schrieb Justin Clark-Casey:
>>>>>>> Regarding the groups work, I have now implemented an
>>>>>>> OpenSimulator experimental option, MessageOnlineUsersOnly in
>>>>>>> [Groups] as of git master 1937e5f. When set to true this will
>>>>>>> only send group IMs to online users. This does not
>>>>>>> require a groups service update. I believe OSGrid is going to
>>>>>>> test this more extensively soon though it appears to
>>>>>>> work fine on Wright Plaza.
>>>>>>>
>>>>>>> It's temporarily a little spammy on the console right now (what
>>>>>>> isn't!) with a debug message that says how many
>>>>>>> online
>>>>>>> users it is sending to and how long a send takes.
>>>>>>>
>>>>>>> Unlike Michelle's solution, this works by querying the Presence
>>>>>>> service for online users, though it also caches this
>>>>>>> data to avoid hitting the presence service too hard.
>>>>>>>
>>>>>>> Even though I implemented this, I'm not convinced that it's the
>>>>>>> best way to go - I think Michelle's approach of
>>>>>>> sending login/logoff status directly from simulator to groups
>>>>>>> service could still be better. My chief concern with
>>>>>>> the groups approach is the potential inconsistency between
>>>>>>> online status stored there and in the Presence service.
>>>>>>> However, this could be a non-issue. Need to give it more thought.
>>>>>>>
>>>>>>> On 14/10/12 22:53, Akira Sonoda wrote:
>>>>>>>> IMHO finding out which group members are online and sending
>>>>>>>> group IM/Notice etc. to them actually should not be
>>>>>>>> done by
>>>>>>>> the region server from which the group IM/notice etc. is sent.
>>>>>>>> This is a task which should be done centrally in case of OSgrid
>>>>>>>> in Dallas TX (
>>>>>>>> http://wiki.osgrid.org/index.php/Infrastructure ). The region
>>>>>>>> server should only collect the group IM/notice etc.
>>>>>>>> and
>>>>>>>> send it to the central group server or in the other way
>>>>>>>> receiving IM/notice etc. from the central group server and
>>>>>>>> distribute it to the Agents active on the region(s).
>>>>>>>
>>>>>>> That concentrates all distribution on a central point rather
>>>>>>> than spreading it amongst simulators. Then OSGrid has
>>>>>>> the problem of scaling this up.
>>>>>>>
>>>>>>> Having said that, there are advantages to funnelling things
>>>>>>> through a reliable central point. As to which is better
>>>>>>> is a complicated engineering issue - the kind of which there are
>>>>>>> many in the MMO/VW space.
>>>>>>>
>>>>>>>>
>>>>>>>> But there are even other places which can and should be
>>>>>>>> improved. I did some tests with some viewers counting the
>>>>>>>> web
>>>>>>>> requests to the central infrastructure:
>>>>>>>>
>>>>>>>> Test 1: Teleport from a Plaza to one of my regions located on a
>>>>>>>> server in Europe and afterwards logging out:
>>>>>>>>
>>>>>>>> Cool VL Viewer: 912 Requests mostly SynchronousRestForms POST
>>>>>>>> http://presence.osgrid.org/presence ( i guess to
>>>>>>>> inform
>>>>>>>> all my 809 friends [mostly only 5% online] I am going offline
>>>>>>>> because the calls to the presence service were done
>>>>>>>> after
>>>>>>>> i closed the viewer)
>>>>>>>> Singularity Veiwer: 921 Requests mostly calls to presence after
>>>>>>>> logoff
>>>>>>>> Teapot viewer: 910 Requests mostly calls to presence after logoff
>>>>>>>> Astra Viewer: 917 Requests mostly calls to presence after logoff
>>>>>>>> Firestorm: 1005 Requests mostly calls to presence after logoff
>>>>>>>> Imprudence: 918 mostly calls to presence after logoff
>>>>>>>>
>>>>>>>> So far so good. I have no idea why my 760 offline friends have
>>>>>>>> to be informed that I went offline ...
>>>>>>>> (Details can be found here:
>>>>>>>> https://docs.google.com/open?id=0B301xueh1kxdNG1wLWo2YVVfYjA )
>>>>>>>>
>>>>>>>> Test 2: Direct Login onto my Region and then Logoff-( with
>>>>>>>> FetchInventory2 disabled )
>>>>>>>>
>>>>>>>> Cool VL Viewer: 2232 Requests mostly calls to presence ~800
>>>>>>>> during login and ~800 during logout and xinventory
>>>>>>>> Singularity Viwer: 2340 Requests mostly calls to presence and
>>>>>>>> xinventory
>>>>>>>> Teapot Viewer: Produced 500+ Threads in a very short time and
>>>>>>>> then the OpenSim.exe crashed
>>>>>>>> Astra Viewer: 2831 Request mostly calls to presence and xinventory
>>>>>>>> Firestorm Viwer: ACK Timeout for me. OpenSim.exe survived on
>>>>>>>> 500 Threads for 30+ minutes producing 4996 Requests
>>>>>>>> mostly
>>>>>>>> xinventory
>>>>>>>> Imprudence: 1745 Requests mostly presence
>>>>>>>>
>>>>>>>> Again why do all my 809 friends have do be verified with single
>>>>>>>> requests? Then why this difference in xinventory
>>>>>>>> Requests? And why are both Teapot and Firestorm producing so
>>>>>>>> many Threads in such a short time? and bring
>>>>>>>> OpenSim.exe to
>>>>>>>> crash or closely to crash ...
>>>>>>>> ( Details can be found here:
>>>>>>>> https://docs.google.com/open?id=0B301xueh1kxdMDJxWm5UR2QtU2c )
>>>>>>>
>>>>>>> The presence information is useful data and it was possible in
>>>>>>> git master commit da2b23f to change the Friends module
>>>>>>> to fetch all presence data in one call for status notification
>>>>>>> when a user goes on/offline, rather than make a
>>>>>>> separate call for each friend.
>>>>>>>
>>>>>>> This should be more efficient since only the latency and
>>>>>>> resources of one call is required. However, since each
>>>>>>> friend still has to be messaged separately to tell them of the
>>>>>>> status change I'm not sure how much practical effect
>>>>>>> this will have.
>>>>>>>
>>>>>>>>
>>>>>>>> Test 3: Direct Login to my Region with FetchInventory2 enabled.
>>>>>>>>
>>>>>>>> Teapot Viewer: I closed the viwer after 30 minutes. Number of
>>>>>>>> Threads were still rising up to 260. In the end i
>>>>>>>> counted
>>>>>>>> 30634 xinventory requests... My Inventory has 14190 items !!!
>>>>>>>> Firestorm Viwer: Quite normal approx 2020 Requests ... quite
>>>>>>>> some slow FetchInventoryDescendandts2 Caps. with 100
>>>>>>>> sec
>>>>>>>> max
>>>>>>>
>>>>>>> Regarding inventory service, unfortunately many viewers appear
>>>>>>> to behave very aggressively when fetching inventory
>>>>>>> information. For instance, I'm told that if you have certain
>>>>>>> types of AO enabled - some viewers will fetch your
>>>>>>> entire inventory. The LL infrastructure may be able to cope
>>>>>>> with this but the more modest machines running grids can
>>>>>>> have trouble, it seems.
>>>>>>>
>>>>>>> I'm not sure what the long term solution is. I suspect it's
>>>>>>> possible to greatly increase inventory fetch efficiency,
>>>>>>> possibly by some kind of call batching. Or perhaps there's some
>>>>>>> viewer-side caching that OpenSimulator isn't working
>>>>>>> with properly.
>>>>>>>
>>>>>>>>
>>>>>>>> ( Details can be found here:
>>>>>>>> https://docs.google.com/open?id=0B301xueh1kxdNEtEeUVFamU1QUE )
>>>>>>>>
>>>>>>>> Just my observations this week end.
>>>>>>>> Akira
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2012/10/13 Justin Clark-Casey <jjustincc at googlemail.com
>>>>>>>> <mailto:jjustincc at googlemail.com>>
>>>>>>>>
>>>>>>>> Hi Michelle. I've now had some more time to think about
>>>>>>>> this. In fact, I established a proposal summary page at
>>>>>>>> [1] which I'll change as we go along (or please feel free
>>>>>>>> to change yourself). We do need to fix this
>>>>>>>> problem of
>>>>>>>> group IM taking massive time with groups that aren't that big.
>>>>>>>>
>>>>>>>> I do like the approach of caching online status (and login
>>>>>>>> time) in the groups service.
>>>>>>>>
>>>>>>>> 1. It's reasonably simple.
>>>>>>>> 2. One network call to fetch online group members per IM.
>>>>>>>> 3. May allow messaging across multiple OpenSimulator
>>>>>>>> installations.
>>>>>>>>
>>>>>>>> However, this approach does mean
>>>>>>>>
>>>>>>>> 1. Independently updating the groups services on each
>>>>>>>> login/logout. I'm not saying this is a problem,
>>>>>>>> particularly
>>>>>>>> if it saves traffic later on.
>>>>>>>> 2. Groups service has to deal with extra information.
>>>>>>>> Again, this is fairly simple so not necessarily a fatal
>>>>>>>> issue though it does mean every groups implementations
>>>>>>>> needs to do this in some manner.
>>>>>>>> 3. Online cache is not reusable by other services in the
>>>>>>>> future.
>>>>>>>>
>>>>>>>> On a technical note, the XmlRpc groups module does in
>>>>>>>> theory cache data for 30 seconds by default, so a
>>>>>>>> change in
>>>>>>>> online status may not be seen for upto 30 seconds. I
>>>>>>>> personally think that this is a reasonable tradeoff.
>>>>>>>>
>>>>>>>> Rather, of the above cons, 3 is the one I'm finding most
>>>>>>>> serious. If other services would also benefit from
>>>>>>>> online
>>>>>>>> status caching in the future, they would have to implement
>>>>>>>> their own caches (and be updated from simulators).
>>>>>>>>
>>>>>>>> I do agree that making a GridUser.LoggedIn() call for every
>>>>>>>> single group member on every single IM is
>>>>>>>> unworkable.
>>>>>>>> Even if this is only done once and cached for a certain
>>>>>>>> period of time it could be a major issue for large
>>>>>>>> groups.
>>>>>>>>
>>>>>>>> So an alternative approach could be to add a new call to
>>>>>>>> GridUser service (maybe LoggedIn(List<UUID>) that will
>>>>>>>> only
>>>>>>>> return GridInfo for those that are logged in. This could
>>>>>>>> then be cached simulator-side for a certain period of
>>>>>>>> time
>>>>>>>> (e.g. 30 seconds like the groups information) and used for
>>>>>>>> group IM.
>>>>>>>>
>>>>>>>> This has the advantages that
>>>>>>>>
>>>>>>>> 1. Groups and future services don't need to do their own
>>>>>>>> login caching.
>>>>>>>> 2. Future services can use the same information and code
>>>>>>>> rather than have to cache login information
>>>>>>>> themselves.
>>>>>>>>
>>>>>>>> However, it does
>>>>>>>>
>>>>>>>> 1. Require GridUserInfo caching simulator-side, I would
>>>>>>>> judge this to be a more complex approach.
>>>>>>>> 2. Mean that during the cache period, new online group
>>>>>>>> messages will not receive messages. (this is going to
>>>>>>>> happen with GetGroupMembers() caching anyway).
>>>>>>>> 3. Traffic is still generated to the GridUser service at
>>>>>>>> the end of every simulator-side caching period.
>>>>>>>> This is
>>>>>>>> probably not a huge burden.
>>>>>>>>
>>>>>>>> So right now, I'm somewhat more in favour of a GridUserInfo
>>>>>>>> simulator-side caching approach than caching login
>>>>>>>> information within the groups service. However, unlike
>>>>>>>> you, I haven't actually tried to implement this
>>>>>>>> approach so
>>>>>>>> there may well be issues that I haven't seen.
>>>>>>>>
>>>>>>>> What do you think, Michelle (or anybody else)?
>>>>>>>>
>>>>>>>>
>>>>>>>> On 10/10/12 19:47, Michelle Argus wrote:
>>>>>>>>
>>>>>>>> http://code.google.com/p/__flotsam/
>>>>>>>> <http://code.google.com/p/flotsam/> is the the current flotsam
>>>>>>>> version and
>>>>>>>> points to the github repro which I forked and
>>>>>>>> then patched.
>>>>>>>>
>>>>>>>> None of the changes I proposed in my git fork have been
>>>>>>>> implemented, neither in opensim nor in flotsam.
>>>>>>>>
>>>>>>>> Consider my proposal as a quick fix for the time
>>>>>>>> beeing which does not solve all other issues
>>>>>>>> mentioned by
>>>>>>>> later
>>>>>>>> mailings.
>>>>>>>>
>>>>>>>> Am 09.10.2012 10:24, schrieb Ai Austin:
>>>>>>>>
>>>>>>>> Michelle Argus on Wed Oct 3 18:00:23 CEST 2012:
>>>>>>>>
>>>>>>>> I have added some changes to the group module
>>>>>>>> of OpenSim and the flotsam server.
>>>>>>>> ...
>>>>>>>> The changes can be found in the 2 gits here:
>>>>>>>> <https://github.com/MAReantals__>https://github.com/MAReantals
>>>>>>>>
>>>>>>>> NB: Both changes to flotsam and opensim are
>>>>>>>> backward compatible and do
>>>>>>>> not require that both parts are updated. If
>>>>>>>> some simulators are not
>>>>>>>> updated it can happen that some groupmembers do
>>>>>>>> not receive
>>>>>>>> groupmessages as their online status is not
>>>>>>>> updated correctly. In a grid
>>>>>>>> like OSgrid my recomendation would thus be to
>>>>>>>> first update the
>>>>>>>> simulators and at a later stage flotsam.
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Michelle... I am looking at what is needed to
>>>>>>>> update the Openvue grid which is using the flotsam
>>>>>>>> XmlRpcGroups
>>>>>>>> module. the GITHub repository has the changes from
>>>>>>>> a few days ago... but I wonder if there has been an
>>>>>>>> update/commit
>>>>>>>> into the main Opensim Github area already. I cannot
>>>>>>>> see a related commit looking back over the last
>>>>>>>> week
>>>>>>>> or so. Is
>>>>>>>> the core system updated so this module is up to
>>>>>>>> date in that? I also note that the Opensim.ini.example
>>>>>>>> file
>>>>>>>> contains
>>>>>>>> a reference to http://code.google.com/p/__flotsam/
>>>>>>>> <http://code.google.com/p/flotsam/> for details of
>>>>>>>> how to
>>>>>>>> install the service.. but that seems to be
>>>>>>>> pointing at an out of date version?
>>>>>>>>
>>>>>>>> I think for the flotsam php end it is
>>>>>>>> straightforward and I obtained the changed groups.sql and
>>>>>>>> xmlrpc.php files
>>>>>>>> needed. But note that people are still pointed via
>>>>>>>> the opensim.ini.example comments at the old
>>>>>>>> version on
>>>>>>>> http://code.google.com/p/__flotsam/
>>>>>>>> <http://code.google.com/p/flotsam/> so that either needs updating
>>>>>>>> to teh
>>>>>>>> latest version, or the comment in
>>>>>>>> opensim.ini.exmaple needs to be changed.
>>>>>>>>
>>>>>>>> To avoid mistakes, I wonder if you can clarify
>>>>>>>> where to go for the parts needed and at what
>>>>>>>> revision/date of
>>>>>>>> OpenSim
>>>>>>>> 0.7.5 dev master this was introduced, what to get
>>>>>>>> and what to change for an existing service in terms
>>>>>>>> of the
>>>>>>>> data base
>>>>>>>> tables, OpenSim.exe instance and the web support
>>>>>>>> php code areas?
>>>>>>>>
>>>>>>>> Thanks Michelle, Ai
>>>>>>>>
>>>>>>>> _________________________________________________
>>>>>>>> Opensim-dev mailing list
>>>>>>>> Opensim-dev at lists.berlios.de
>>>>>>>> <mailto:Opensim-dev at lists.berlios.de>
>>>>>>>> https://lists.berlios.de/__mailman/listinfo/opensim-dev
>>>>>>>> <https://lists.berlios.de/mailman/listinfo/opensim-dev>
>>>>>>>>
>>>>>>>>
>>>>>>>> _________________________________________________
>>>>>>>> Opensim-dev mailing list
>>>>>>>> Opensim-dev at lists.berlios.de
>>>>>>>> <mailto:Opensim-dev at lists.berlios.de>
>>>>>>>> https://lists.berlios.de/__mailman/listinfo/opensim-dev
>>>>>>>> <https://lists.berlios.de/mailman/listinfo/opensim-dev>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Justin Clark-Casey (justincc)
>>>>>>>> OSVW Consulting
>>>>>>>> http://justincc.org
>>>>>>>> http://twitter.com/justincc
>>>>>>>> _________________________________________________
>>>>>>>> Opensim-dev mailing list
>>>>>>>> Opensim-dev at lists.berlios.de
>>>>>>>> <mailto:Opensim-dev at lists.berlios.de>
>>>>>>>> https://lists.berlios.de/__mailman/listinfo/opensim-dev
>>>>>>>> <https://lists.berlios.de/mailman/listinfo/opensim-dev>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Opensim-dev mailing list
>>>>>>>> Opensim-dev at lists.berlios.de
>>>>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Opensim-dev mailing list
>>>>>> Opensim-dev at lists.berlios.de
>>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Opensim-dev mailing list
>>>> Opensim-dev at lists.berlios.de
>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>
>
More information about the Opensim-dev
mailing list