MantisBT - opensim
View Issue Details
0008455opensim[GRID] Hypergridpublic2019-01-16 01:462019-02-01 09:00
Grid (Multiple Regions per Sim)
.NET / Windows64
0008455: Child agents not closed cleanly when HG traveller exits grid - closes only after 60000ms no packets received timeout
When a HG traveller left the grid and was (I believe) properly returned to their home grid, all the adjacent regions on two separate servers showed yellow messages after 5 minutes (60000ms) timeout... such as...

23:37:29 - [LLUDPSERVER]: No packets received from child agent of Brian.. for 60000ms in Edinburgh Uplands. Disconnecting.

There have been changes to Dev Master in recent days in the area of child agent clean up for the teleport handling. However, the Git commits indicate that HG teleports are handled in the old way at present. It is possible this issue has been present for longer, but I am specifically monitoring teleport attempts due to the code changes and thought it worth opening this Mantis issue to allow inputs on any similar issues from those using Dev master versions after the [2019-01-14 17:25] commit.
Full example log... avatar left grid at 22:53:41 then no console messages or activity until 60000ms (approx.) timeout at 23:40:50.

22:53:40 - [HG ENTITY TRANSFER MODULE]: Teleporting agent via landmark to [^] region 33f15d55-b426-4e70-a942-26606cbe49cf position <121.6659, 119.8572, 22.16383>
22:53:40 - [GATEKEEPER SERVICE CONNECTOR]: contacting [^]
22:53:40 - [ENTITY TRANSFER MODULE]: Teleporting Brian.. af4d1b2d-1427-4aef-9234-4a57077873e9 from Openvue to [^] ( [^]) hrethra/<121.6659, 119.8572, 22.16383>
22:53:40 - [REMOTE SIMULATION CONNECTOR]: QueryAccess to [^] returned True, reason , version 0.7/0.7
22:53:40 - [ENTITY TRANSFER MODULE]: Openvue transfer protocol version to hrethra is 0.7 / 0.7
22:53:40 - [ENTITY TRANSFER MODULE]: Determined that region hrethra at 8654,8654 size 256,256 needs new child agent for agent Brian.. from Openvue
22:53:40 - [HG ENTITY TRANSFER MODULE]: CreateAgent [^] [^]
22:53:40 - [REMOTE SIMULATION CONNECTOR]: Creating agent at [^]
22:53:40 - [ENTITY TRANSFER MODULE]: Sending new CAPS seed url [^] from Openvue to Brian..
22:53:41 - [SCENE PRESENCE]: Making Brian.. a child agent in Openvue from root region 9515173628988928
23:40:50 - [LLUDPSERVER]: No packets received from child agent of Brian.. for 60000ms in Sandbox. Disconnecting.
23:40:50 - [CLIENT]: Close has been called for Brian.. attached to scene Sandbox
23:40:50 - [JobEngine]: Stopping AsyncInUDP-af4d1b2d-1427-4aef-9234-4a57077873e9
23:40:50 - [SCENE]: Removing child agent Brian.. af4d1b2d-1427-4aef-9234-4a57077873e9 from Sandbox
23:40:50 - [CAPS]: Remove caps for agent af4d1b2d-1427-4aef-9234-4a57077873e9 in region Sandbox
23:40:50 - [Scene]: The avatar has left the building
No tags attached.
txt 2019-02-01-OSGrid-TP-Issue.txt (9,973) 2019-02-01 09:00
Issue History
2019-01-16 01:46aiaustinNew Issue
2019-01-16 01:47aiaustinSummaryChild agents not closed cleanly when HG traveller exits grid - closes only after 60000ms no packets received timeout. => Child agents not closed cleanly when HG traveller exits grid - closes only after 60000ms no packets received timeout
2019-01-16 01:47aiaustinDescription Updatedbug_revision_view_page.php?rev_id=7547#r7547
2019-01-16 01:51aiaustinDescription Updatedbug_revision_view_page.php?rev_id=7548#r7548
2019-01-16 01:51aiaustinAdditional Information Updatedbug_revision_view_page.php?rev_id=7550#r7550
2019-01-16 05:26tampaNote Added: 0033819
2019-01-16 05:28aiaustinNote Added: 0033820
2019-01-16 07:24aiaustinDescription Updatedbug_revision_view_page.php?rev_id=7551#r7551
2019-01-18 07:32aiaustinNote Added: 0033833
2019-02-01 05:49aiaustinNote Added: 0034092
2019-02-01 05:49aiaustinNote Edited: 0033833bug_revision_view_page.php?bugnote_id=33833#r7738
2019-02-01 05:51aiaustinNote Edited: 0034092bug_revision_view_page.php?bugnote_id=34092#r7740
2019-02-01 05:52aiaustinNote Edited: 0034092bug_revision_view_page.php?bugnote_id=34092#r7741
2019-02-01 07:12aiaustinNote Edited: 0034092bug_revision_view_page.php?bugnote_id=34092#r7742
2019-02-01 08:59aiaustinNote Added: 0034093
2019-02-01 09:00aiaustinFile Added: 2019-02-01-OSGrid-TP-Issue.txt
2019-02-01 09:03aiaustinNote Edited: 0034093bug_revision_view_page.php?bugnote_id=34093#r7744
2019-02-01 09:03aiaustinNote Edited: 0034093bug_revision_view_page.php?bugnote_id=34093#r7745
2019-02-01 09:05aiaustinNote Edited: 0034093bug_revision_view_page.php?bugnote_id=34093#r7746
2019-02-01 09:06aiaustinNote Edited: 0034093bug_revision_view_page.php?bugnote_id=34093#r7747
2019-02-01 09:09aiaustinNote Edited: 0034093bug_revision_view_page.php?bugnote_id=34093#r7748
2019-02-01 09:10aiaustinNote Edited: 0034093bug_revision_view_page.php?bugnote_id=34093#r7749
2019-02-01 09:16aiaustinNote Edited: 0034093bug_revision_view_page.php?bugnote_id=34093#r7750

2019-01-16 05:26   
Forgive me for saying, but maybe it is time for teleportsV2. This back and forth with timeouts and small patches here and there does not really seems very scientific nor problem orientated. Without knowing specifically if this is, again, a viewer thing maybe it really is time to handle all this sort of stuff over http/tcp where possible, because it just occurs to me that the vast majority of connection and teleport issues arise from slightly less perfect connections and people using wireless.

It just seems like we are fighting symptoms of a bigger issue rather than fixing the actual issue underneath. Just my opinion though.
2019-01-16 05:28   
Perhaps Ubit’s new teleport protocol version 0.7 is that?
2019-01-18 07:32   
(edited on: 2019-02-01 05:49)
Child agents remain on the

15:33:18 - [LLUDPSERVER]: No packets received from child agent of Ai.Austin for 60000ms in OpenVCE. Disconnecting.

So HG travellers are definitely not being tidied up for 60000ms and only then due to an inactivity timeout. The root agent has been tidied up, so I assume the information is available server side to tidy up.

2019-02-01 05:49   
(edited on: 2019-02-01 07:12)
I am seeing more TP issues even on one grid that appears to follow any failure to TP between regions which are running but seem not to latch onto the TP. After the failure, I seem to be stuck in a mode where I have lots of child agents (on regions that were attempted but failed TP targets and are definitely more than draw distance way regions) and no root agent. Viewer (eventually) returns avatar to original region before TP attempt, but canno tmove and need to relog. The child agents seem to stay in the target regions which were being tried and "show users full" shows them as still present for a long time afterwards (way more than 60000ms as I left things longer than that to test if they timed out). The child agents in region consoles only disappeared after AFTER the viewer was stopped.

I observed this red error on the target region on one TP attempt if it helps...

13:43:30 - [BASE HTTP SERVER]: HandleRequest() threw exception System.NullReferenceException: Object reference not set to an instance of an object.
   at OpenSim.Region.Framework.Scenes.SceneObjectPart.get_GroupPosition() in K:\OSGRID\opensim-\OpenSim\Region\Framework\Scenes\SceneObjectPart.cs:line 764
   at OpenSim.Region.Framework.Scenes.SceneObjectGroup.set_AbsolutePosition(Vector3 value) in K:\OSGRID\opensim-\OpenSim\Region\Framework\Scenes\SceneObjectGroup.cs:line 600
   at OpenSim.Region.CoreModules.Avatar.Attachments.AttachmentsModule.AttachToAgent(IScenePresence sp, SceneObjectGroup so, UInt32 attachmentpoint, Vector3 attachOffset, Boolean silent) in K:\OSGRID\opensim-\OpenSim\Region\CoreModules\Avatar\Attachments\AttachmentsModule.cs:line 1072
   at OpenSim.Region.CoreModules.Avatar.Attachments.AttachmentsModule.AttachObjectInternal(IScenePresence sp, SceneObjectGroup group, UInt32 attachmentPt, Boolean silent, Boolean addToInventory, Boolean resumeScripts, Boolean append) in K:\OSGRID\opensim-\OpenSim\Region\CoreModules\Avatar\Attachments\AttachmentsModule.cs:line 654
   at OpenSim.Region.CoreModules.Avatar.Attachments.AttachmentsModule.AttachObject(IScenePresence sp, SceneObjectGroup group, UInt32 attachmentPt, Boolean silent, Boolean addToInventory, Boolean append) in K:\OSGRID\opensim-\OpenSim\Region\CoreModules\Avatar\Attachments\AttachmentsModule.cs:line 541
   at OpenSim.Region.Framework.Scenes.Scene.AddSceneObject(SceneObjectGroup sceneObject) in K:\OSGRID\opensim-\OpenSim\Region\Framework\Scenes\Scene.cs:line 2960
   at OpenSim.Region.CoreModules.Framework.EntityTransfer.EntityTransferModule.HandleIncomingSceneObject(SceneObjectGroup so, Vector3 newPosition) in K:\OSGRID\opensim-\OpenSim\Region\CoreModules\Framework\EntityTransfer\EntityTransferModule.cs:line 2693
   at OpenSim.Region.CoreModules.Framework.EntityTransfer.HGEntityTransferModule.HandleIncomingSceneObject(SceneObjectGroup so, Vector3 newPosition) in K:\OSGRID\opensim-\OpenSim\Region\CoreModules\Framework\EntityTransfer\HGEntityTransferModule.cs:line 647
   at OpenSim.Region.Framework.Scenes.Scene.IncomingCreateObject(Vector3 newPosition, ISceneObject sog) in K:\OSGRID\opensim-\OpenSim\Region\Framework\Scenes\Scene.cs:line 2889
   at OpenSim.Region.CoreModules.Avatar.Attachments.AttachmentsModule.CopyAttachments(AgentData ad, IScenePresence sp) in K:\OSGRID\opensim-\OpenSim\Region\CoreModules\Avatar\Attachments\AttachmentsModule.cs:line 356
   at OpenSim.Region.Framework.Scenes.ScenePresence.CopyFrom(AgentData cAgent) in K:\OSGRID\opensim-\OpenSim\Region\Framework\Scenes\ScenePresence.cs:line 4859
   at OpenSim.Region.Framework.Scenes.ScenePresence.UpdateChildAgent(AgentData cAgentData) in K:\OSGRID\opensim-\OpenSim\Region\Framework\Scenes\ScenePresence.cs:line 4638
   at OpenSim.Region.Framework.Scenes.Scene.IncomingUpdateChildAgent(AgentData cAgentData) in K:\OSGRID\opensim-\OpenSim\Region\Framework\Scenes\Scene.cs:line 4545
   at OpenSim.Region.CoreModules.ServiceConnectorsOut.Simulation.LocalSimulationConnectorModule.UpdateAgent(GridRegion destination, AgentData cAgentData, EntityTransferContext ctx) in K:\OSGRID\opensim-\OpenSim\Region\CoreModules\ServiceConnectorsOut\Simulation\LocalSimulationConnector.cs:line 218
   at OpenSim.Server.Handlers.Simulation.AgentPutHandler.UpdateAgent(GridRegion destination, AgentData agent) in K:\OSGRID\opensim-\OpenSim\Server\Handlers\Simulation\AgentHandlers.cs:line 766
   at OpenSim.Server.Handlers.Simulation.AgentPutHandler.DoAgentPut(Hashtable request, Hashtable responsedata) in K:\OSGRID\opensim-\OpenSim\Server\Handlers\Simulation\AgentHandlers.cs:line 735
   at OpenSim.Server.Handlers.Simulation.AgentPutHandler.ProcessRequest(String path, Stream request, IOSHttpRequest httpRequest, IOSHttpResponse httpResponse) in K:\OSGRID\opensim-\OpenSim\Server\Handlers\Simulation\AgentHandlers.cs:line 668
   at OpenSim.Framework.Servers.HttpServer.BaseStreamHandler.Handle(String path, Stream request, IOSHttpRequest httpRequest, IOSHttpResponse httpResponse) in K:\OSGRID\opensim-\OpenSim\Framework\Servers\HttpServer\BaseStreamHandler.cs:line 72
   at OpenSim.Framework.Servers.HttpServer.BaseHttpServer.HandleRequest(OSHttpRequest request, OSHttpResponse response) in K:\OSGRID\opensim-\OpenSim\Framework\Servers\HttpServer\BaseHttpServer.cs:line 694

2019-02-01 08:59   
(edited on: 2019-02-01 09:16)
Maybe I should shift the "on one grid" TP issue to another Mantis.. but it just seems to have very similar symptoms to do with missing a connect between a TP to a region even within one grid and then it getting in a mess with leaving child agents around and not allowing TP back and jamming up the avatar after the timeouts. The terrain and water is also missing.. Remember this is between two regions within ONE grid (OSGrid) both on 26-Jan-2019) version.. no HG involved.

I attach a full log of the OSGrid regions TP attempt (Vue-Port to VRLand both on the same server in this test in different openSim.exe instances).


The main issue seems to be that I appear to connect (Completing movement) and then the next log line is 10 seconds later when it says it did not receive agent update. After that its jammed u, avatar on original region cannot move, and child agents remain on destination region console using "show users full" until relog. I left it well over 5 minutes to see if it timed out after 60000ms and it did not.

16:55:39 - [SCENE PRESENCE]: Completing movement of Ai Austin into region VRLand in position <128, 128, 1.5>
16:55:49 - [SCENE PRESENCE]: Update agent Ai Austin at VRLand did not receive agent update

After about 17 minutes, with the avatar still stuck on the original source region and unable to move, etc, I logged out of the viewer.. then after about another minute I get the no packets received message... but this did NOT happen until the viewer was stopped.

17:12:17 - [LLUDPSERVER]: No packets received from child agent of Ai Austin for 60000ms in VRLand. Disconnecting.
17:12:17 - [CLIENT]: Close has been called for Ai Austin attached to scene VRLand.

On logging back I the viewer usually thinks I am still logged in.. and it needs a second attempt.