Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008672opensim[REGION] Unhandled Packetspublic2020-03-08 00:382020-04-03 03:05
ReporterWaki Janus 
Assigned To 
PriorityhighSeveritymajorReproducibilityalways
StatusnewResolutionopen 
PlatformMac Book ProOSCatalina 10.15.3OS VersionYeti
Product Version0.9.1.0 
Target VersionFixed in Version 
Summary0008672: Motion Comms Broken
DescriptionThis occurs when we have a large number of people on a big region with a complex builds and physics. It usually takes 30 - 60 minutes to occur. The result is avatars freeze and cannot move. Everyone can continue to chat, but otherwise can do nothing else such as move, TP or create notes etc. Occasionally it results in viewer crashes which throw up a error messages that say things like "serious mangled data errors".
Steps To ReproduceAs above.
Additional InformationKitto Flora maybe able to describe this better as I am not a tech.
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (Multiple Regions per Sim)
Physics EngineubODE
Script EngineYEngine
EnvironmentMono / OSX
Mono VersionNone
ViewerFirestorm 6.02 (for OS)
Attached Files

- Relationships

-  Notes
(0036270)
tampa (reporter)
2020-03-08 22:31

It sounds more like you are actively overloading the region or have it misconfigured to actually handle the load you are putting on to it. There is no magic here eventually, given enough physics and scripts, regions grind to a halt. Take a look at the performance meter and see if Simfps are going down. Either throw more hardware at the problem or change some configuration parameters to better deal with heavy loads.
(0036271)
Waki Janus (reporter)
2020-03-09 21:21

No it is not that. We are not actively overloading the region or have it misconfigured. I suggest you look at Issue report 0008666 lodged by Kitto Flora. We think this was or is the same problem, but different manifestation resulting in a breakdown in motion comms between viewers and OS. We had about 20 people on the region when it occurred. All were affected though they were using different viewers or versions of the viewers. It not a resources problem as several of them have heavy duty gaming machines. However, having said that we installed the new update that Ubit has released and testing it on a copy of the region affected. So far after running nearly 4 hours with trains and other vehicles etc running there seems no issues as described by Kitto Flora in 0008666. So the Ubit fix seems to be working well.
(0036274)
Waki Janus (reporter)
2020-03-10 16:55

Further to the above. Kitto Flora was on Dreamnation with me when a single sim running Ubit's latest fix on Yeti (Our 092b) went through repeated crashes. He has reported this at 0008666 ((0036273). But, I'll repeat what he reported:
"Update and an error:
We now have OpenSim 0.9.2.0 Yeti DreamNation 910aced58a4e421fb981f8afa82afefd71d0cecf 2020-03-09 06:03:52 -0700 (Unix/Mono)
which was built Monday early morning (USA Pacific time) and is locally called '092B'.
I have since had two 3+ hour sessions on the big region with no UDP errors.
092B has also been loaded onto a smaller region, with a 6+ Avatars on it. It has had several crashes and self-restarts. Extract from the simulator console capture:
14:30:16 - [APPLICATION]:
APPLICATION EXCEPTION DETECTED: System.UnhandledExceptionEventArgs

Exception: System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at ....
....
Application is terminating: True

Region (Phoebus) # [ERROR] FATAL UNHANDLED EXCEPTION: System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at ....

....
14:30:22 - ========================= STARTING OPENSIM =========================

I have the sim console messages saved if you need more detail.

Note for tampa: these regions are running on serious Dell servers: 20CPU cores, typically 80% idle, KiB Mem : 19804574+total, 11115732+free, 21647212 used, 65241208 buff/cache."

This is a variation on what I previously reported above. It is clear that this is a variation on what I reported originally above and that the bug has got worse with the latest update of Yeti.

I would also suggest that someone other than Tampa responds to this.
(0036276)
tampa (reporter)
2020-03-11 00:19

> I would also suggest that someone other than Tampa responds to this.

Rudeness ain't gonna get you anywhere.

As I posted in the other ticket without more complete logs it is hard to determine the source module of the error. We need to know where to look.

Without access to your environment to debug directly all we can attempt to do is to reproduce the issue on our end and attempt to narrow down the cause. It also doesn't matter how much money you are wasting on an idling server either, hardware and software limitations still exists and specifically for OpenSim more cores =/= better performance, that is not to say that is the cause here.

I suspect there is something on that region that triggers an edge case that isn't properly handled, but to determine the trigger is difficult without information on how to reproduce it or at least knowing where the failure happens(to give the code a review and perhaps attempt to handle what isn't handled). Given there are multiple avatars and probably scripts or physics going on you have a lot of variables and potentials for disaster. If you can narrow it down to "if this avatar teleport" or "if that script is triggered" that would help quite a bit.

YEngine does have some iffy behavior when it encounters grave errors like scripts running out of heap, so whenever I see errors the first thing I jump to is running the region without scripts or even with just basicphysics. If the problem stops you have your culprit, then it becomes a question of figuring out which script causes that and what it does that makes Y barf.
(0036322)
UbitUmarov (administrator)
2020-04-03 03:05

Please try to provide more detailed logs. To know that a Collection was modified is not very helpful. There should be more information than than

- Issue History
Date Modified Username Field Change
2020-03-08 00:38 Waki Janus New Issue
2020-03-08 22:31 tampa Note Added: 0036270
2020-03-09 21:21 Waki Janus Note Added: 0036271
2020-03-10 16:55 Waki Janus Note Added: 0036274
2020-03-11 00:19 tampa Note Added: 0036276
2020-04-03 03:05 UbitUmarov Note Added: 0036322


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker