[Opensim-dev] Update for group module & flotsam

Sun Oct 14 21:53:06 UTC 2012

IMHO finding out which group members are online and sending group IM/Notice
etc. to them actually should not be done by the region server from which
the group IM/notice etc. is sent.
This is a task which should be done centrally in case of OSgrid in Dallas
TX ( http://wiki.osgrid.org/index.php/Infrastructure ). The region server
should only collect the group IM/notice etc. and send it to the central
group server or in the other way receiving IM/notice etc. from the central
group server and distribute it to the Agents active on the region(s).

But there are even other places which can and should be improved. I did
some tests with some viewers counting the web requests to the central
infrastructure:

Test 1: Teleport from a Plaza to one of my regions located on a server in
Europe and afterwards logging out:

Cool VL Viewer: 912 Requests mostly SynchronousRestForms POST
http://presence.osgrid.org/presence ( i guess to inform all my 809 friends
[mostly only 5% online] I am going offline because the calls to the
presence service were done after i closed the viewer)
Singularity Veiwer: 921 Requests mostly calls to presence after logoff
Teapot viewer: 910 Requests mostly calls to presence after logoff
Astra Viewer: 917 Requests mostly calls to presence after logoff
Firestorm: 1005 Requests mostly calls to presence after logoff
Imprudence: 918 mostly calls to presence after logoff

So far so good. I have no idea why my 760 offline friends have to be
informed that I went offline ...
(Details can be found here:
https://docs.google.com/open?id=0B301xueh1kxdNG1wLWo2YVVfYjA )

Test 2: Direct Login onto my Region and then Logoff-( with FetchInventory2
disabled )

Cool VL Viewer: 2232 Requests mostly calls to presence ~800 during login
and ~800 during logout and xinventory
Singularity Viwer: 2340 Requests mostly calls to presence and xinventory
Teapot Viewer: Produced 500+ Threads in a very short time and then the
OpenSim.exe crashed
Astra Viewer: 2831 Request mostly calls to presence and xinventory
Firestorm Viwer: ACK Timeout for me. OpenSim.exe survived on 500 Threads
for 30+ minutes producing 4996 Requests mostly xinventory
Imprudence: 1745 Requests mostly presence

Again why do all my 809 friends have do be verified with single requests?
Then why this difference in xinventory Requests? And why are both Teapot
and Firestorm producing so many Threads in such a short time? and bring
OpenSim.exe to crash or closely to crash ...
( Details can be found here:
https://docs.google.com/open?id=0B301xueh1kxdMDJxWm5UR2QtU2c )

Test 3: Direct Login to my Region with FetchInventory2 enabled.

Teapot Viewer: I closed the viwer after 30 minutes. Number of Threads were
still rising up to 260. In the end i counted 30634 xinventory requests...
My Inventory has 14190 items !!!
Firestorm Viwer: Quite normal approx 2020 Requests ... quite some slow
FetchInventoryDescendandts2 Caps. with 100 sec max

( Details can be found here:
https://docs.google.com/open?id=0B301xueh1kxdNEtEeUVFamU1QUE )

Just my observations this week end.
Akira

2012/10/13 Justin Clark-Casey <jjustincc at googlemail.com>

> Hi Michelle.  I've now had some more time to think about this.  In fact, I
> established a proposal summary page at [1] which I'll change as we go along
> (or please feel free to change yourself).  We do need to fix this problem
> of group IM taking massive time with groups that aren't that big.
>
> I do like the approach of caching online status (and login time) in the
> groups service.
>
> 1.  It's reasonably simple.
> 2.  One network call to fetch online group members per IM.
> 3.  May allow messaging across multiple OpenSimulator installations.
>
> However, this approach does mean
>
> 1.  Independently updating the groups services on each login/logout.  I'm
> not saying this is a problem, particularly if it saves traffic later on.
> 2.  Groups service has to deal with extra information.  Again, this is
> fairly simple so not necessarily a fatal issue though it does mean every
> groups implementations needs to do this in some manner.
> 3.  Online cache is not reusable by other services in the future.
>
> On a technical note, the XmlRpc groups module does in theory cache data
> for 30 seconds by default, so a change in online status may not be seen for
> upto 30 seconds.  I personally think that this is a reasonable tradeoff.
>
> Rather, of the above cons, 3 is the one I'm finding most serious.  If
> other services would also benefit from online status caching in the future,
> they would have to implement their own caches (and be updated from
> simulators).
>
> I do agree that making a GridUser.LoggedIn() call for every single group
> member on every single IM is unworkable.  Even if this is only done once
> and cached for a certain period of time it could be a major issue for large
> groups.
>
> So an alternative approach could be to add a new call to GridUser service
> (maybe LoggedIn(List<UUID>) that will only return GridInfo for those that
> are logged in.  This could then be cached simulator-side for a certain
> period of time (e.g. 30 seconds like the groups information) and used for
> group IM.
>
> This has the advantages that
>
> 1.  Groups and future services don't need to do their own login caching.
> 2.  Future services can use the same information and code rather than have
> to cache login information themselves.
>
> However, it does
>
> 1.  Require GridUserInfo caching simulator-side, I would judge this to be
> a more complex approach.
> 2.  Mean that during the cache period, new online group messages will not
> receive messages.  (this is going to happen with GetGroupMembers() caching
> anyway).
> 3.  Traffic is still generated to the GridUser service at the end of every
> simulator-side caching period.  This is probably not a huge burden.
>
> So right now, I'm somewhat more in favour of a GridUserInfo simulator-side
> caching approach than caching login information within the groups service.
>  However, unlike you, I haven't actually tried to implement this approach
> so there may well be issues that I haven't seen.
>
> What do you think, Michelle (or anybody else)?
>
>
> On 10/10/12 19:47, Michelle Argus wrote:
>
>> http://code.google.com/p/**flotsam/ <http://code.google.com/p/flotsam/>is the the current flotsam version and points to the github repro which I
>> forked and
>> then patched.
>>
>> None of the changes I proposed in my git fork have been implemented,
>> neither in opensim nor in flotsam.
>>
>>   Consider my proposal as a quick fix for the time beeing which does not
>> solve all other issues mentioned by later
>> mailings.
>>
>> Am 09.10.2012 10:24, schrieb Ai Austin:
>>
>>> Michelle Argus on Wed Oct 3 18:00:23 CEST 2012:
>>>
>>>> I have added some changes to the group module of OpenSim and the
>>>> flotsam server.
>>>> ...
>>>> The changes can be found in the 2 gits here: <
>>>> https://github.com/MAReantals**>https://github.com/MAReantals
>>>>
>>>> NB: Both changes to flotsam and opensim are backward compatible and do
>>>> not require that both parts are updated. If some simulators are not
>>>> updated it can happen that some groupmembers do not receive
>>>> groupmessages as their online status is not updated correctly. In a grid
>>>> like OSgrid my recomendation would thus be to first update the
>>>> simulators and at a later stage flotsam.
>>>>
>>>
>>> Hi Michelle... I am looking at what is needed to update the Openvue grid
>>> which is using the flotsam XmlRpcGroups
>>> module.  the GITHub repository has the changes from a few days ago...
>>> but I wonder if there has been an update/commit
>>> into the main Opensim Github area already.  I cannot see a  related
>>> commit looking back over the last week or so.  Is
>>> the core system updated so this module is up to date in that?  I also
>>> note that the Opensim.ini.example file contains
>>> a reference to http://code.google.com/p/**flotsam/<http://code.google.com/p/flotsam/>for details of how to install the service.. but that seems to be
>>> pointing at an out of date version?
>>>
>>> I think for the flotsam php end it is straightforward and I obtained the
>>> changed groups.sql and xmlrpc.php files
>>> needed.  But note that people are still pointed via the
>>> opensim.ini.example comments at the old version on
>>> http://code.google.com/p/**flotsam/ <http://code.google.com/p/flotsam/>so that either needs updating to teh latest version, or the comment in
>>> opensim.ini.exmaple needs to be changed.
>>>
>>> To avoid mistakes, I wonder if you can clarify where to go for the parts
>>> needed and at what revision/date of OpenSim
>>> 0.7.5 dev master this was introduced, what to get and what to change for
>>> an existing service in terms of the data base
>>> tables, OpenSim.exe instance and the web support php code areas?
>>>
>>> Thanks Michelle, Ai
>>>
>>> ______________________________**_________________
>>> Opensim-dev mailing list
>>> Opensim-dev at lists.berlios.de
>>> https://lists.berlios.de/**mailman/listinfo/opensim-dev<https://lists.berlios.de/mailman/listinfo/opensim-dev>
>>>
>>>
>> ______________________________**_________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/**mailman/listinfo/opensim-dev<https://lists.berlios.de/mailman/listinfo/opensim-dev>
>>
>>
>
> --
> Justin Clark-Casey (justincc)
> OSVW Consulting
> http://justincc.org
> http://twitter.com/justincc
> ______________________________**_________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/**mailman/listinfo/opensim-dev<https://lists.berlios.de/mailman/listinfo/opensim-dev>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20121014/2283cb5e/attachment-0001.html>