Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007308opensim[REGION] OpenSim Corepublic2014-08-19 14:132015-07-24 17:13
Assigned To 
PlatformMonoOSUbuntuOS Version
Product Versionmaster (dev code) 
Target VersionFixed in Version 
Summary0007308: Network Lag caused by specific users connection
DescriptionI'm having problems when a specific user connects to my grid. When she connects, she has major lag problems which are then also spread to all the other users on the region. She has tried different viewers and accounts and also regions on two separate servers on our grid. As the problem persists the only common variable in all instances is her connection. The console command 'show queues' indicates high numbers for resent packets and unacked packets (Screen shot included).

It seems very strange that a bad connection (if thats what it is) should affect other users so severely. The last time she was on, the console showed a lot of the following messages and then crashed with no other messages printed to the debug file..

OpenSim.Region.ClientStack.LindenUDP.LLClientView [CLIENT]: Unhandled packet ViewerEffect from Cara Crystal (root ) in Honeywood. Ignoring.
OpenSim.Region.ClientStack.LindenUDP.LLClientView [CLIENT]: Unhandled packet AgentCachedTexture from Cara Crystal (root ) in Honeywood. Ignoring.

TagsNo tags attached.
Git Revision or version number
Run Mode Grid (1 Region per Sim)
Physics EngineBulletSim
Script Engine
EnvironmentMono / Linux64
Mono Version3.6
ViewerFirestorm and Singularity
Attached Filesgif file icon network.gif [^] (6,464 bytes) 2014-08-19 14:13

- Relationships
related to 0007393closedadministrator Visitors with high packet latency cause region to experience excessive lag and other performance issues 

-  Notes
justincc (administrator)
2014-08-19 16:30

That is a very high number of resends that indicates either a viewer that's not processing them properly (which shouldn't happen if a common viewer is used) or a very very bad connection.

But yes, resends shouldn't cause a problem in this way. However, from the load tests as well I'm beginning to wonder if there might not be some 'crowding out' bug where a backlogged packet queue for one connection is preventing timely sending of other connection packets. The code is structured not to allow this to happen but perhaps there is some bug. This will be one of my focus areas in the next few weeks.

What when Cara disconnects - does sim response return to normal for other users?

Also, if Cara is using wireless has she tried a wired connection? Some wireless is fine but I've encountered some issues in the past where a low quality or poorly configured wireless access point has led to major packet loss with the rapid UDP exchange that OpenSimulator requires.
justincc (administrator)
2014-08-19 16:37

The unhandled packet messages are odd, though. ViewerEffect, for instance, is one inbound packet that LLClientView is definitely set to handle.

On login, do the Pkts In climb rapidly for Cara also climb much more rapidly than one would normally expect? If there are a huge number of resends downstream one might also expect the viewer to be generating lots of resends the other way.

Finally, it would also be interesting to see what throttle ends up being set for Cara, whether due to packet loss OpenSimulator has massively adjusted it downwards.
AliciaRaven (manager)
2014-08-19 16:59

After she disconnects the sim takes a few minutes to get back to normal. I think about 5 minutes after she left i went back in and things were back to the standard performance levels. Those dropped packet messages were not being displayed at first, they appeared on her last visit and where soon followed by the sim crashing. I think it will be hard to get more data as I'm not sure when she will return, and after the shocking performance tonight she may have given up.

What other server commands should i use apart from the show queues?

When she was in the region i clicked a box to buy its contents and i got no response at all, other than my avatar sending the selection beam effect and the box showing it was highlighted. It seems when she is there, others are unable to do even simple tasks in world. She has visited once before a few months ago and she had no problems then.

I looked in the grid service logs as well and most of her login requests where followed by a slow json-rpc request, example bellow.

.LLLoginService [LLOGIN SERVICE]: Login request for Cara Crystal at last using viewer Singularity, channel Singularity, IP *****, Mac *****, Id0 *****
.WebUtil [LOGHTTP]: Slow JSON-RPC request 3 POST to http://Grid_IP:9000/agent/***/ [^] took 6184ms, 5ms writing, {"agent_id":"***","caps_path":"***","circuit_code":"***","first_name":"Cara","last_name":"Crystal","secure_session_id":"***

As a final note, the viewers she was using i can see from the log where a bit out of date, i don't think they were old enough to cause issue but here are the versions incase Firestorm-Release and Singularity

Thanks :)
justincc (administrator)
2014-12-16 12:43

Beyond trying to replicate this problem manually, which I don't know when I'll get the opportunity to do, one way to try and get a better handle on this is to use the stats recording facility to capture information about sim behaviour over time when this problem is occurring [1] that we can then analyze to look for problems.

This is best done on master if possible. may still yield data but it will be difficult to add new stats as I suspect existing ones may not be enough.

The stat recording files get very large so will need to be posted somewhere else but in my experience recording the data has no significant impact on sim performance.

[1] [^]
Diva (administrator)
2015-07-24 16:59

Does this bug still exist, or did it go away with the inventory improvements?
Mata Hari (reporter)
2015-07-24 17:13

@Diva: This bug still happens, although it would seem somewhat less prone to occurring than was the case prior to your inventory work -- perhaps by virtue of reducing the bottleneck a little? A single viewer with high latency can still bring a region to its knees (that happened not too long ago in Wright Plaza during a dev meeting). See also 0007393 for more data on it.

- Issue History
Date Modified Username Field Change
2014-08-19 14:13 AliciaRaven New Issue
2014-08-19 14:13 AliciaRaven File Added: network.gif
2014-08-19 16:30 justincc Note Added: 0026772
2014-08-19 16:37 justincc Note Added: 0026773
2014-08-19 16:59 AliciaRaven Note Added: 0026774
2014-12-15 21:41 AliciaRaven Relationship added related to 0007393
2014-12-16 12:43 justincc Note Added: 0027112
2015-07-24 16:59 Diva Note Added: 0028966
2015-07-24 17:13 Mata Hari Note Added: 0028967

Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker