Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007037opensim[GRID] Messaging Servicepublic2014-02-25 16:422014-05-22 07:16
Reporterhack13 
Assigned Tojustincc 
PrioritynormalSeveritymajorReproducibilityalways
Statuspatch feedbackResolutionreopened 
PlatformMonoOSLinuxOS Version0.7.6-Post-Fixes
Product Version 
Target VersionFixed in Versionmaster (dev code) 
Summary0007037: Group Notices Cause Time Outs (Both V2 and Flostream)
DescriptionWhen sending a group notice to any group with more than 100 users it causes the region to time out all the avatars that where their during the notice being sent.
Steps To Reproduce1. Find a group with over 100 people
2. Login at least 2 avatars in the same region
3. Send a notice to that group
Additional InformationThis happens in both V2 and the Flostream
TagsNo tags attached.
Git Revision or version number0.7.6-Post-Fixes
Run Mode Grid (1 Region per Sim)
Physics EngineODE
Script Engine
EnvironmentMono / Linux64
Mono Version2.10
Viewer
Attached Files

- Relationships

-  Notes
(0025318)
Snoopy (administrator)
2014-02-26 00:47

I can confirm this old bug. This seems to happen when the GridUsers database table has the state *logged in* for some group members that in fact are logged out or have crashed.
(0025321)
usagabe (reporter)
2014-02-26 07:41

I can confirm this happens to us as well, it is happening a lot and causes people to then not able to get back in to the region until we reboot the region. It happens when there are a lot of people in the group and it seems to cause a viewer timeout for everyone on the sim. This happens with Singularity Firestorm & other viewers e.g Imprudence and Cool
(0025339)
hack13 (reporter)
2014-03-02 14:36

So a little more detail I have found watching the console. It seems to freeze updating all avatars in the simulator, then they time out(crash). Then it starts sending the notices.
(0025381)
justincc (administrator)
2014-03-06 17:45

Does this also happen when members simply message each other?

I suspect the problem here is that sending a group notice is effectively the same as other group IM as it's fanned out to many recipients, but it's not using the changes put in last year for ordinary messages.
(0025383)
hack13 (reporter)
2014-03-06 18:51
edited on: 2014-03-06 18:51

No just group notices, whatever Diva did before fixed the IMing lag but not the notices.

(0025406)
justincc (administrator)
2014-03-10 17:39

Okay, in git master 77e7bbc I have routed group notice sends through the same mechanism as other message sends. This should help with the delay issues, though you may well need to have MessageOnlineUsersOnly = true set in [Groups] (a setting which should probably now be default).

I've only implemented this for XmlRpcGroups for now in case there are any issues. Is anybody in a position to test? If not with dev, then after a short while I will backport the fix to 0.7.6-post-fixes. At some point after, I will make the same change to V2 groups.
(0025408)
hack13 (reporter)
2014-03-10 18:01

I have a few grids that would be happy to test it. We will test and push back results to you.
(0025422)
hack13 (reporter)
2014-03-11 15:30

Ok so we don't crash anymore however it doesn't send the notice to anyone, here is what the console says.

[GROUPS-MESSAGING]: SendMessageToGroup for group
cac0b831-140b-4819-bdbf-4806b34ab7fb with 0 visible members, 0 online
took 0ms
18:22:25 - [GROUPS-MESSAGING]: SendMessageToGroup for group
cac0b831-140b-4819-bdbf-4806b34ab7fb with 0 visible members, 0 online
took 0ms
18:22:58 - [USER MANAGEMENT MODULE]: No grid user found for
da73ca36-5c77-47d0-b922-1dbbd8cd6acb
18:22:58 - [USER MANAGEMENT MODULE]: No grid user found for
ea2a566d-46f2-4a80-a06a-efef2fb4e41a
18:23:00 - [GROUPS-MESSAGING]: SendMessageToGroup for group
cac0b831-140b-4819-bdbf-4806b34ab7fb with 0 visible members, 0 online
took 0ms
(0025432)
usagabe (reporter)
2014-03-12 08:23

We can confirm the same thing happens to us groups functionality breaks altogether. And sending out a notice is the same type of errors 0 visible member and 0 online.
(0025437)
justincc (administrator)
2014-03-12 16:27

Please try git master beba208 - there was an oversight which I have now fixed and it should work.

However, there might be an occasional issue where the groups module does not always find the IGroupsMessagingModule implementation on startup even though it should always be registered by the time the search happens. Please report if you see the warning

[GROUPS]: Could not get IGroupsMessagingModule

in the logs.
(0025441)
hack13 (reporter)
2014-03-13 11:47

This patch appears to be holding really strong so far, we will report back in a few days if it is still holding up.
(0025446)
hack13 (reporter)
2014-03-14 12:43

Reporting that it is still working flawlessly. But we are still testing.
(0025591)
hack13 (reporter)
2014-03-26 10:16

Ok, new issue. Now groups notices/messages don't appear to get saved/sent to users who are offline now.
(0025599)
justincc (administrator)
2014-03-26 16:37

Are you using the offline IM module in OpenSimulator itself?
(0025600)
hack13 (reporter)
2014-03-26 16:41

We are using php based
(0025601)
justincc (administrator)
2014-03-26 16:42

Did you set MessageOnlineUsersOnly = true for this issue as described above or were you already using this setting?
(0025608)
hack13 (reporter)
2014-03-26 19:18

yes we were, and it appears to still be working on older region on the grid but the regions we are running this test code on it is not.
(0025926)
hack13 (reporter)
2014-04-29 06:12

Ok, update on this ticket. This has resolved the region crashing issue, however now the group notices do not save to offline users can get them. This poses a another problem.
(0025947)
justincc (administrator)
2014-05-01 14:40

Yes, when we set MessageOnlineUsersOnly = true, the simulator no longer attempts to send the notices to offline avatars. This means IMessageTransferModule.OnUndeliveredMessage is no longer triggered, which is what was triggering the send to the OfflineMessageURL configured in the [Messaging] part of OpenSim.ini.

However, the original timeouts were very probably triggered by precisely this action. It's possible that something in the async messaging on the old path was the cause of the timeouts, but even then I can't help thinking that for large groups, posing 900 requests to that URL in quick succession (assuming 900 offline users) is not going to be a quick process.

So ultimately, a different approach might be better, such as sending a single HTTP request with a single notice but a list of the 900 offline user UUIDs that should receive that notice, rather than 900 separate requests for each offline user. But that's going to be a change that needs to be thought out carefully, and then only after we're sure that the root problem isn't something to do with the async message sending instead (though even then, the current approach looks very inefficient).

With what size user groups were you seeing this problem? Do you have an idea what proportion of users were offline?
(0025952)
hack13 (reporter)
2014-05-01 21:03

Pretty much any group that has over 50-60 member experience this issue. I have also noticed that the issue is not just isolated to floatstam groups but also the built in v2.
(0026001)
usagabe (reporter)
2014-05-09 11:58

Wanted to add to this as well, could you please push this up this fix to put in 0.8.0 stable.

Our users no longer crash when sending out notices to a group of 302 members where as it used to be.

However, now only users when logged in are getting the group notices.
If a user is offline they are not getting any more even to email. Or if they were logged off when it was sent they don't get it either.

Hope this information helps solve this one and we can get a fix for that last issue.

Thank you kindly for all the help and support.
(0026055)
justincc (administrator)
2014-05-12 15:54

Ok, please try git master 8ad29fc as soon as possible. This restores the triggering of IMessageTransferModule.OnUndeliveredMessage when MessageOnlineUsersOnly = true, which is necessary for offline IM to be saved.

I don't believe that this will suffer from the same scene freezing problems as previously (when MessageOnlineUsersOnly = false) since it doesn't try massive parallel sending of messages or undeliverable notifications. However, I cannot be absolutely sure of this, so testing with your problem groups is critical.

This only currently applies to the Flotsam/XmlRpcGroups. Once I know this works, it should be possible to extend it to core groups as well.

Also, is there any public PHP code for storing/retrieving the offline messages sent to the URL?
(0026066)
kcozens (administrator)
2014-05-13 09:11
edited on: 2014-05-13 10:24

In reply to Justin's question in comment 0026055 (above), storing and retrieving offline messages when using Flotsam/XmlRpcGroups can be handled using the offline.php file which is part of the OpenSim Web Interface project in Forge. The URL for that project is http://forge.opensimulator.org/gf/project/opensimwi/ [^]

(0026084)
justincc (administrator)
2014-05-13 16:44
edited on: 2014-05-13 16:44

When testing, please try the later git master c23d37a instead, since I had to workaround an existing bug which the offline IM URL POST timeout change brought to light.

(0026103)
hack13 (reporter)
2014-05-14 13:26

100% Patched :D
(0026104)
justincc (administrator)
2014-05-14 13:33

Just reopening so that I remember to make the same change for core groups soon.
(0026112)
hack13 (reporter)
2014-05-15 18:24

Two new issues:
1. no more ability to send attachments in the notices
2. it takes up to 4 minutes to send a notice on just a group with 2 people
(0026128)
justincc (administrator)
2014-05-19 15:07

1. Attachment issue fixed in git master 3a6f312. This was a recent regression.

2. I'm going to need much more information since I don't see that issue on my systems. In git master b46be88 I have separated the groups and group messaging debug flags so one can get more information without being overwhelmed (though there's still a lot of text).

So on XmlRpcGroups, when you see this issue, please enable with

"debug groups messaging verbose true".

This will post various information including time taken to send the message and breakdown of different send times. Please post the results.
(0026164)
usagabe (reporter)
2014-05-22 07:16

Will test this out. Thanks

- Issue History
Date Modified Username Field Change
2014-02-25 16:42 hack13 New Issue
2014-02-26 00:47 Snoopy Note Added: 0025318
2014-02-26 07:41 usagabe Note Added: 0025321
2014-03-02 14:36 hack13 Note Added: 0025339
2014-03-06 17:44 justincc Assigned To => justincc
2014-03-06 17:44 justincc Status new => assigned
2014-03-06 17:45 justincc Note Added: 0025381
2014-03-06 18:51 hack13 Note Added: 0025383
2014-03-06 18:51 hack13 Note Edited: 0025383 View Revisions
2014-03-10 17:39 justincc Note Added: 0025406
2014-03-10 17:39 justincc Status assigned => feedback
2014-03-10 18:01 hack13 Note Added: 0025408
2014-03-10 18:01 hack13 Status feedback => assigned
2014-03-11 15:30 hack13 Note Added: 0025422
2014-03-12 02:13 danbanner Note Added: 0025431
2014-03-12 02:22 danbanner Note Deleted: 0025431
2014-03-12 08:23 usagabe Note Added: 0025432
2014-03-12 16:27 justincc Note Added: 0025437
2014-03-12 16:27 justincc Status assigned => feedback
2014-03-13 11:47 hack13 Note Added: 0025441
2014-03-13 11:47 hack13 Status feedback => assigned
2014-03-14 12:43 hack13 Note Added: 0025446
2014-03-26 10:16 hack13 Note Added: 0025591
2014-03-26 16:37 justincc Note Added: 0025599
2014-03-26 16:41 hack13 Note Added: 0025600
2014-03-26 16:42 justincc Note Added: 0025601
2014-03-26 19:18 hack13 Note Added: 0025608
2014-04-29 06:12 hack13 Note Added: 0025926
2014-05-01 14:40 justincc Note Added: 0025947
2014-05-01 21:03 hack13 Note Added: 0025952
2014-05-09 11:58 usagabe Note Added: 0026001
2014-05-12 15:54 justincc Note Added: 0026055
2014-05-13 09:11 kcozens Note Added: 0026066
2014-05-13 09:47 smxy Note Added: 0026067
2014-05-13 10:24 kcozens Note Edited: 0026066 View Revisions
2014-05-13 11:45 smxy Note Deleted: 0026067
2014-05-13 16:44 justincc Note Added: 0026084
2014-05-13 16:44 justincc Note Edited: 0026084 View Revisions
2014-05-14 13:26 hack13 Note Added: 0026103
2014-05-14 13:26 hack13 Status assigned => resolved
2014-05-14 13:26 hack13 Fixed in Version => master (dev code)
2014-05-14 13:26 hack13 Resolution open => fixed
2014-05-14 13:33 justincc Note Added: 0026104
2014-05-14 13:33 justincc Status resolved => feedback
2014-05-14 13:33 justincc Resolution fixed => reopened
2014-05-14 13:33 justincc Status feedback => assigned
2014-05-15 18:24 hack13 Note Added: 0026112
2014-05-15 18:24 hack13 Status assigned => resolved
2014-05-15 18:24 hack13 Status resolved => patch feedback
2014-05-19 15:07 justincc Note Added: 0026128
2014-05-22 07:16 usagabe Note Added: 0026164


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker