Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008382opensim[GRID] Grid Servicepublic2018-09-26 13:042018-10-11 10:20
Reportermac4kent 
Assigned To 
PrioritynormalSeverityminorReproducibilitysometimes
StatusnewResolutionopen 
PlatformOSOS Version
Product Version0.9.0.1 
Target VersionFixed in Version 
Summary0008382: TP some time fail
DescriptionI am show you on youtube about TP some time fail.

https://www.youtube.com/watch?v=yGMe7NWteLs&feature=youtu.be [^]

I am use Macintosh.
MacOS 10.14
Firestorm version 5.1.7.55786 (5.1.7.55786)

TagsNo tags attached.
Git Revision or version number
Run Mode Standalone (Multiple Regions)
Physics EngineOther
EnvironmentUnknown
Mono VersionNone
ViewerFirestorm 5.1.7.55786
Attached Filestxt file icon 2018-10-09-ailand-teleport-issue-logs.txt [^] (24,031 bytes) 2018-10-09 09:41 [Show Content]

- Relationships

-  Notes
(0033099)
mewtwo0641 (reporter)
2018-09-28 09:11

I see this issue quite often on my own server and I'm not sure why. There seems to be roughly a 50/50 chance (By my completely unscientific estimate lol) of it working or not and once it decides not to work, nothing except a re-log fixes it, but only very temporarily.

Symptoms of it happening include drifting off into the void when region crossing, eternally black teleport screens, no feed back in the OpenSim console except for the initial message stating a crossing/teleport was initiated, etc. and about 30 seconds later you're either returned to the originating region or disconnected with an error message that connection was lost.

It happens so often that it's actually almost just easier to re-log first before even attempting a region cross/teleport.

It's a perplexing issue because it happens on both LAN (OS is running local to my network) and on WAN.
(0033106)
aiaustin (developer)
2018-10-01 02:27
edited on: 2018-10-09 10:50

I have been seeing something like this between Windows 10 grids (Openvue and AiLand) and to and from my OSGrid addon regions (also on a windows 10 system) for a while now and I can't pin things down yet. So just making a note here in case others are spotting something amiss.

Sometimes even on idling grids/region servers my first attempt to teleport fails after 30 seconds or so and leaves me in the original region and then a second attempt works. Sometimes on second attempt it says I am logged in already in destination (when I clearly aren't there) and then works on the third attempt, other times it crashes the viewer (Firestorm 5.1.7.55786). Destination grid console logs indicate it sees the connection attempt but then not much else happens before the viewer fails. So I am assuming its getting a timeout and not syncing up correctly even on servers that are otherwise idle. @Ubit did recently amend the number of connection attempts and shortened the time between them, but this problem occurs with the original and modified attempts timings.

But as I say I have not pinned anything specific down yet even though its been happening for a few months with latest dev 0.9.1 master (and the httptests branch before that was merged in).

(0033155)
aiaustin (developer)
2018-10-09 09:41
edited on: 2018-10-09 12:09

To try to get a start on pinning down what might be happening.. I am attaching a log file with a few examples of what the OpenSim.exe logs look like on my experimental AiLand grid setup which has one server with Robust.exe, the MySQL 5.7 data base and one OpenSim.exe instance running three regions, including the "home" default region. The second server runs one OpenSim,.exe instance with a number of regions. I have issues for some months and cannot pin things down. All using dev master latest as it gets released.

I get MANY problems TPing between regions, back to the home region, etc.. even without any Hypergrid jumps. Sometimes it fails on the first attempt and then a second attempt immediately works. Sometimes the avatar seems to get stuck and does not show as logged into the origin or destination region and may not be able to move.

User in Firestorm 5.1.7.55786 Windows 10 64 bit viewer sees a message that there are problems connecting between viewer and server each time. I am using a viewer on the same local network as the servers and also over the Internet from home and that does not seem to matter.

Attached log file (2018-10-09-ailand-teleport-issue-logs.txt) with some of the OpenSim.exe logs showing a number of connection problems... these are on idling servers and its not clear why timeouts are occurring or attempts do not work always.

I do see a couple of exceptions on teleport such as

[ENTITY TRANSFER MODULE]: Exception on teleport of Ai Austin from <128.9578, 127.8008, 26.06984>@AiLand to <213, 146, 103.5>@Castle: Object reference not set to an instance of an object.

and

[BASE HTTP SERVER]: HandleRequest() threw exception System.NullReferenceException: Object reference not set to an instance of an object.

(0033156)
watcher64 (reporter)
2018-10-10 23:02
edited on: 2018-10-10 23:03

Been testing those a lot, those errors are not the cause but are a symptom, when you start the TP, you sort of get there, and it fails but it immediately tries again it seems, but some of your attachments are already/still in the scene and it blows right up ..

Been chasing this error and I have something mitigates the error but does not solve the underlying issue ... So very hesitant to share it ..

Just out of curiosity are you running Gloebits on the region?

(0033157)
watcher64 (reporter)
2018-10-10 23:05
edited on: 2018-10-10 23:06

@aiaustin, you can see what I was talking about in the previous post in one line of the log ..

[SCENEGRAPH]: Scene graph for AiLand already contains object 611f510e-b0d3-4e18-9a1d-2dc47d71cb1f in AddSceneObject()
BASE HTTP SERVER]: HandleRequest() threw exception System.NullReferenceException: Object reference not set to an instance of an object.
   at OpenSim.Region.Framework.Scenes.SceneObjectPart.get_GroupPosition() in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\Framework\Scenes\SceneObjectPart.cs:line 817
   at OpenSim.Region.Framework.Scenes.SceneObjectGroup.set_AbsolutePosition(Vector3 value) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\Framework\Scenes\SceneObjectGroup.cs:line 582
   at OpenSim.Region.CoreModules.Avatar.Attachments.AttachmentsModule.AttachToAgent(IScenePresence sp, SceneObjectGroup so, UInt32 attachmentpoint, Vector3 attachOffset, Boolean silent) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\CoreModules\Avatar\Attachments\AttachmentsModule.cs:line 990
   at OpenSim.Region.CoreModules.Avatar.Attachments.AttachmentsModule.AttachObjectInternal(IScenePresence sp, SceneObjectGroup group, UInt32 attachmentPt, Boolean silent, Boolean addToInventory, Boolean resumeScripts, Boolean append) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\CoreModules\Avatar\Attachments\AttachmentsModule.cs:line 578
   at OpenSim.Region.CoreModules.Avatar.Attachments.AttachmentsModule.AttachObject(IScenePresence sp, SceneObjectGroup group, UInt32 attachmentPt, Boolean silent, Boolean addToInventory, Boolean append) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\CoreModules\Avatar\Attachments\AttachmentsModule.cs:line 465
   at OpenSim.Region.Framework.Scenes.Scene.AddSceneObject(SceneObjectGroup sceneObject) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\Framework\Scenes\Scene.cs:line 2965
   at OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule.HandleIncomingSceneObject(SceneObjectGroup so, Vector3 newPosition) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\CoreModules\Framework\EntityTransfer\EntityTransferModule.cs:line 2649
   at OpenSim.Region.CoreModules.Framework.EntityTransfer.HGEntityTransferModule.HandleIncomingSceneObject(SceneObjectGroup so, Vector3 newPosition) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\CoreModules\Framework\EntityTransfer\HGEntityTransferModule.cs:line 598
   at OpenSim.Region.Framework.Scenes.Scene.IncomingCreateObject(Vector3 newPosition, ISceneObject sog) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\Framework\Scenes\Scene.cs:line 2894
   at OpenSim.Region.CoreModules.Avatar.Attachments.AttachmentsModule.CopyAttachments(AgentData ad, IScenePresence sp) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\CoreModules\Avatar\Attachments\AttachmentsModule.cs:line 278
   at OpenSim.Region.Framework.Scenes.ScenePresence.CopyFrom(AgentData cAgent) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\Framework\Scenes\ScenePresence.cs:line 4856
   at OpenSim.Region.Framework.Scenes.ScenePresence.UpdateChildAgent(AgentData cAgentData) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\Framework\Scenes\ScenePresence.cs:line 4635
   at OpenSim.Region.Framework.Scenes.Scene.IncomingUpdateChildAgent(AgentData cAgentData) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\Framework\Scenes\Scene.cs:line 4558
   at OpenSim.Region.CoreModules.ServiceConnectorsOut.Simulation.LocalSimulationConnectorModule.UpdateAgent(GridRegion destination, AgentData cAgentData, EntityTransferContext ctx) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Region\CoreModules\ServiceConnectorsOut\Simulation\LocalSimulationConnector.cs:line 218
   at OpenSim.Server.Handlers.Simulation.AgentPutHandler.UpdateAgent(GridRegion destination, AgentData agent) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Server\Handlers\Simulation\AgentHandlers.cs:line 766
   at OpenSim.Server.Handlers.Simulation.AgentPutHandler.DoAgentPut(Hashtable request, Hashtable responsedata) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Server\Handlers\Simulation\AgentHandlers.cs:line 735
   at OpenSim.Server.Handlers.Simulation.AgentPutHandler.ProcessRequest(String path, Stream request, IOSHttpRequest httpRequest, IOSHttpResponse httpResponse) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Server\Handlers\Simulation\AgentHandlers.cs:line 668
   at OpenSim.Framework.Servers.HttpServer.BaseStreamHandler.Handle(String path, Stream request, IOSHttpRequest httpRequest, IOSHttpResponse httpResponse) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Framework\Servers\HttpServer\BaseStreamHandler.cs:line 72
   at OpenSim.Framework.Servers.HttpServer.BaseHttpServer.HandleRequest(OSHttpRequest request, OSHttpResponse response) in d:\Temp\opensim-0.9.0.1-468-g235dd37\OpenSim\Framework\Servers\HttpServer\BaseHttpServer.cs:line 706

(0033160)
aiaustin (developer)
2018-10-11 01:25
edited on: 2018-10-11 01:30

My grids are not using Gloebits.

My normal, non region owning, users also report the TP problems, with sometimes the second attempt working, and also report significantly slower region contents download to what they experienced previously. I see very slow loading of region content too, even when the regions have been up a while to let any startup tasks, map tile creation, etc, complete. And slow even when nothing else is happening on the server side.

Not sure when this started but I have a niggling worry it may have been when the last major Windows 10 update to winver 1803 was installed. I cannot be sure about that though, and I am reluctant to downgrade to test if that is a factor. This also coincided with the time we moved to .net4.6 (initially on the httptests branch which I adopted in April) and other changes.

(0033161)
aiaustin (developer)
2018-10-11 02:06
edited on: 2018-10-11 02:06

@watcher64, compared to what you were seeing in say February/March 2018 do you see slow down in loading of region content (prims, meshes and textures) along with the TP failures.. or am I perhaps looking at a separate problem there?

(0033162)
mac4kent (reporter)
2018-10-11 09:20

Now all of you know OpenSIM 0.8.3 and OpenSIM 0.9.1 happen.
https://www.osgrid.org/ [^]
(0033163)
watcher64 (reporter)
2018-10-11 09:20

I mainly run linux regions but , I did notice my windows connected test regions were running a bit slower ..

(as far object loading there was a big change to master regarding this a few days ago, but it does not seem to affect this sporadic tp issue)

I have suspected a Windows Update / Anti-virus combo causing issues for a while but the idea did not get much traction .
(0033164)
aiaustin (developer)
2018-10-11 10:20
edited on: 2018-10-11 10:24

I am only using standard Windows Defender and no third party anti-virus package. Problem did start some months ago... April 2018 or so, and latest update from dev master has no effect on issue.

Slow load I note is for region content, rather than inventory. But lets focus in this issue on the faild TP problems and if necessary I will start a separate Mantis issue for slow content load once that is fixed or settles down.


- Issue History
Date Modified Username Field Change
2018-09-26 13:04 mac4kent New Issue
2018-09-28 09:11 mewtwo0641 Note Added: 0033099
2018-10-01 02:27 aiaustin Note Added: 0033106
2018-10-01 02:28 aiaustin Note Edited: 0033106 View Revisions
2018-10-01 02:29 aiaustin Note Edited: 0033106 View Revisions
2018-10-01 03:02 aiaustin Note Edited: 0033106 View Revisions
2018-10-09 09:41 aiaustin Note Added: 0033155
2018-10-09 09:41 aiaustin File Added: 2018-10-09-ailand-teleport-issue-logs.txt
2018-10-09 09:43 aiaustin Note Edited: 0033155 View Revisions
2018-10-09 10:34 aiaustin Note Edited: 0033155 View Revisions
2018-10-09 10:50 aiaustin Note Edited: 0033106 View Revisions
2018-10-09 12:09 aiaustin Note Edited: 0033155 View Revisions
2018-10-10 23:02 watcher64 Note Added: 0033156
2018-10-10 23:03 watcher64 Note Edited: 0033156 View Revisions
2018-10-10 23:05 watcher64 Note Added: 0033157
2018-10-10 23:06 watcher64 Note Edited: 0033157 View Revisions
2018-10-11 01:25 aiaustin Note Added: 0033160
2018-10-11 01:27 aiaustin Note Edited: 0033160 View Revisions
2018-10-11 01:30 aiaustin Note Edited: 0033160 View Revisions
2018-10-11 02:06 aiaustin Note Added: 0033161
2018-10-11 02:06 aiaustin Note Edited: 0033161 View Revisions
2018-10-11 09:20 mac4kent Note Added: 0033162
2018-10-11 09:20 watcher64 Note Added: 0033163
2018-10-11 10:20 aiaustin Note Added: 0033164
2018-10-11 10:22 aiaustin Note Edited: 0033164 View Revisions
2018-10-11 10:24 aiaustin Note Edited: 0033164 View Revisions


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker