[Opensim-users] ReleaseOSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23shutdown problems

Thomas Ringate tringate at gmail.com
Thu Nov 26 21:09:31 UTC 2015


Hi,

I see there is no way to actually reopen a bug in mantis, but I did post some additional information to thes bug in mantis.

Here is the latest thing I posted.

More information on this bug.

To say windows with .net is working correctly is also false.  It is indeed throwing an ERROR  at shutdown and could well be what is causing mono to hang.

Here is a shutdown in windows using .net for the second to the last version of opensim for OSgrid, and the last version of opensim released for OSgrid.

You can clearly see there are no ERRORS in shutdown previously, and there is after the current release.

There is something wrong that was not wrong before in mono and in .net.  The difference is that .NET still closes, and mono does not.

OSgrid 0.8.3.0 (Dev) 41b2855: 2015-10-21

2015-11-26 15:46:32,310 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing all threads
2015-11-26 15:46:32,310 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing listener thread
2015-11-26 15:46:32,310 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing clients
2015-11-26 15:46:32,310 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing console and terminating
2015-11-26 15:46:32,310 INFO  (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Closing down the single simulator: Development
2015-11-26 15:46:32,357 DEBUG (Heartbeat-(Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Heartbeat-(Development), ID 67
2015-11-26 15:46:32,420 DEBUG (Maintenance (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Maintenance (Development), ID 72
2015-11-26 15:46:32,826 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Persisting changed objects
2015-11-26 15:46:37,029 DEBUG (1) - OpenSim.Services.GridService.GridService [GRID SERVICE]: Deregistering region Development (570c1600-3516-11e4-8c21-0800200c9a66) at 10054-9986
2015-11-26 15:46:37,295 DEBUG (1) - OpenSim.OpenSimBase [SHUTDOWN]: Shutting down region Development
2015-11-26 15:46:37,310 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_chan_search.php?cond_channame=570c1600-3516-11e4-8c21-0800200c9a66D&auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235>
2015-11-26 15:46:37,466 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice]: region Development: uuid 570c1600-3516-11e4-8c21-0800200c9a66: located directory id 4443169
2015-11-26 15:46:37,482 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_chan_search.php?&cond_chanparent=4443169&auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235>
2015-11-26 15:46:37,576 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_chan_mod.php?mode=delete&chan_id=4443169&auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235>
2015-11-26 15:46:37,717 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: Region Development is being removed, removing from indexing
2015-11-26 15:46:37,748 INFO  (1) - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Shutting down 492 scripts in Development
2015-11-26 15:46:37,842 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread0, ID 58
2015-11-26 15:46:37,857 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread1, ID 59
2015-11-26 15:46:37,857 INFO  (1) - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Shutting down the LLUDP server for Development
2015-11-26 15:46:37,857 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping outbound UDP loop
2015-11-26 15:46:37,857 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping inbound UDP loop
2015-11-26 15:46:37,904 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_signout.php?auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235>
2015-11-26 15:46:37,967 DEBUG (Outgoing Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Outgoing Packets (Development), ID 62
2015-11-26 15:46:37,967 DEBUG (Incoming Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Incoming Packets (Development), ID 61
2015-11-26 15:46:37,982 DEBUG (bulletunmanaged (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread bulletunmanaged (Development), ID 57
2015-11-26 15:46:37,998 INFO  (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.SimianGroupsServicesConnectorModule [SIMIAN-GROUPS-CONNECTOR]: Closing SimianGroupsServicesConnector
2015-11-26 15:46:37,998 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.XmlRpcGroupsServicesConnectorModule [XMLRPC-GROUPS-CONNECTOR]: Closing XmlRpcGroupsServicesConnector
2015-11-26 15:46:38,092 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: data service http://search.osgrid.org/register.php notified. Secret: 24383411-9692-40df-b8f1-e22e2bb9aa45
2015-11-26 15:46:38,107 INFO  (1) - OpenSim.Framework.Servers.BaseOpenSimServer [SHUTDOWN]: Shutdown processing on main thread complete.  Exiting...


OSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23

2015-11-26 15:49:02,654 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing all threads
2015-11-26 15:49:02,654 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing listener thread
2015-11-26 15:49:02,654 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing clients
2015-11-26 15:49:02,654 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing console and terminating
2015-11-26 15:49:02,654 INFO  (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Closing down the single simulator: Development
2015-11-26 15:49:03,170 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: TriggerSceneShuttingDown
2015-11-26 15:49:03,170 INFO  (1) - OpenSim.Region.Framework.Scenes.EventManager [EVENT MANAGER]: TriggerSceneShuttingDown done
2015-11-26 15:49:03,170 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Persisting changed objects
2015-11-26 15:49:03,186 DEBUG (Maintenance (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Maintenance (Development), ID 77
2015-11-26 15:49:03,217 DEBUG (36) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Heartbeat-(Development), ID 72
2015-11-26 15:49:03,217 ERROR (36) - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat-(Development)". ThreadState=Stopped. Last tick was 625ms ago.  
2015-11-26 15:49:03,295 DEBUG (TouchAllSceneAssets) - OpenSim.Region.CoreModules.Asset.FlotsamAssetCache [FLOTSAM ASSET CACHE]: Could not find asset ce5301d9-0943-4ffa-891e-f8305b748897, type 0 referenced by object @-Heather Genie Dressing Booth at <48.14737, 75.16174, 22.52853> in scene Development when pre-caching all scene assets
2015-11-26 15:49:23,795 DEBUG (1) - OpenSim.Services.GridService.GridService [GRID SERVICE]: Deregistering region Development (570c1600-3516-11e4-8c21-0800200c9a66) at 10054-9986
2015-11-26 15:49:24,061 DEBUG (1) - OpenSim.OpenSimBase [SHUTDOWN]: Shutting down region Development
2015-11-26 15:49:24,280 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice]: region Development: uuid 570c1600-3516-11e4-8c21-0800200c9a66: located directory id 4443177
2015-11-26 15:49:24,686 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: Region Development is being removed, removing from indexing
2015-11-26 15:49:24,733 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread0, ID 64
2015-11-26 15:49:24,749 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread1, ID 65
2015-11-26 15:49:24,749 INFO  (1) - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Shutting down 492 scripts in Development
2015-11-26 15:49:24,827 INFO  (1) - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Shutting down the LLUDP server for Development
2015-11-26 15:49:24,827 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping outbound UDP loop
2015-11-26 15:49:24,827 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping inbound UDP loop
2015-11-26 15:49:24,842 DEBUG (MapItemRequestThread (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread MapItemRequestThread (Development), ID 58
2015-11-26 15:49:24,858 DEBUG (Outgoing Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Outgoing Packets (Development), ID 68
2015-11-26 15:49:24,858 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Dispose Physics
2015-11-26 15:49:24,889 DEBUG (Incoming Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Incoming Packets (Development), ID 67
2015-11-26 15:49:24,952 DEBUG (bulletunmanaged (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread bulletunmanaged (Development), ID 57
2015-11-26 15:49:25,014 INFO  (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.SimianGroupsServicesConnectorModule [SIMIAN-GROUPS-CONNECTOR]: Closing SimianGroupsServicesConnector
2015-11-26 15:49:25,014 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.XmlRpcGroupsServicesConnectorModule [XMLRPC-GROUPS-CONNECTOR]: Closing XmlRpcGroupsServicesConnector
2015-11-26 15:49:25,093 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: data service http://search.osgrid.org/register.php notified. Secret: d73bfc15-b3db-46b6-800c-d5d320185478
2015-11-26 15:49:25,108 INFO  (1) - OpenSim.Framework.Servers.BaseOpenSimServer [SHUTDOWN]: Shutdown processing on main thread complete.  Exiting...

As  I said, I have no proof this is the mono problem, but there is something wrong and it sure is suspicious that it might well be what is causing mono to hang and not close.

From: AJLDuarte 
Sent: Wednesday, November 25, 2015 11:11 PM
To: opensim-users at opensimulator.org 
Subject: Re: [Opensim-users] ReleaseOSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23shutdown problems

Hi

                Similar issue was already reported, and found to be related to mono

                Check http://opensimulator.org/mantis/view.php?id=7743

regards

 

From: opensim-users-bounces at opensimulator.org [mailto:opensim-users-bounces at opensimulator.org] On Behalf Of Han Held
Sent: Thursday, November 26, 2015 03:59
To: opensim-users at opensimulator.org
Subject: Re: [Opensim-users] ReleaseOSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23 shutdown problems

 

More likely than not, it's a mono problem. I've encountered that sporadically over the years. What is your os (vendor, version, 32 bit or 64 bit) and what's the output of mono --version ?

On 11/25/15 1:39 PM, Thomas Ringate wrote:

  Anyone else having a problem with shutting down a simulator since upgrading to release OSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23?

   

  I can only shut down the simulator by using a Linux “kill” command.  The opensim “quit” command no longer works.  It goes to the point you get the message “Exiting...” and then hangs forever.

   

  The simulators appear to be running fine, but kind of a nuisance not to be able to use normal commands to shut them down.

   

  They are running in a “screen” terminal window.

   

  Here is the log of a shutdown immediately after launching a region.

   

  2015-11-25 17:18:29,952 INFO  (1) - OpenSim.Framework.Servers.ServerBase [SERVER BASE]: Running shutdown_commands.txt

  2015-11-25 17:18:29,978 INFO  (1) - OpenSim.Framework.Servers.ServerBase [SERVER BASE]: Running 'backup'

  2015-11-25 17:18:33,925 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing all threads

  2015-11-25 17:18:33,926 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing listener thread

  2015-11-25 17:18:33,926 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing clients

  2015-11-25 17:18:33,926 INFO  (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing console and terminating

  2015-11-25 17:18:33,927 INFO  (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Closing down the single simulator: Sanctuary

  2015-11-25 17:18:34,148 DEBUG (Maintenance (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Maintenance (Sanctuary), ID 79

  2015-11-25 17:18:34,428 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: TriggerSceneShuttingDown

  2015-11-25 17:18:34,428 INFO  (1) - OpenSim.Region.Framework.Scenes.EventManager [EVENT MANAGER]: TriggerSceneShuttingDown done

  2015-11-25 17:18:34,429 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Persisting changed objects

  2015-11-25 17:18:34,588 DEBUG (1) - OpenSim.Services.GridService.GridService [GRID SERVICE]: Deregistering region Sanctuary (3b15c050-1fe4-11df-8a39-0800200c9a66) at 10054-9986

  2015-11-25 17:18:34,909 DEBUG (1) - OpenSim.OpenSimBase [SHUTDOWN]: Shutting down region Sanctuary

  2015-11-25 17:18:35,678 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice]: region Sanctuary: uuid 3b15c050-1fe4-11df-8a39-0800200c9a66: located directory id 4439471

  2015-11-25 17:18:35,773 DEBUG (Threadpool worker) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Heartbeat-(Sanctuary), ID 75

  2015-11-25 17:18:35,774 ERROR (Threadpool worker) - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat-(Sanctuary)". ThreadState=Stopped. Last tick was 1916ms ago.  

  2015-11-25 17:18:36,025 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: Region Sanctuary is being removed, removing from indexing

  2015-11-25 17:18:36,031 INFO  (1) - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Shutting down 487 scripts in Sanctuary

  2015-11-25 17:18:36,208 INFO  (1) - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Shutting down the LLUDP server for Sanctuary

  2015-11-25 17:18:36,208 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping outbound UDP loop

  2015-11-25 17:18:36,209 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping inbound UDP loop

  2015-11-25 17:18:36,210 DEBUG (Outgoing Packets (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Outgoing Packets (Sanctuary), ID 59

  2015-11-25 17:18:36,223 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread0, ID 66

  2015-11-25 17:18:36,223 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread1, ID 67

  2015-11-25 17:18:36,231 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Dispose Physics

  2015-11-25 17:18:36,234 DEBUG (bulletunmanaged (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread bulletunmanaged (Sanctuary), ID 57

  2015-11-25 17:18:36,246 DEBUG (Incoming Packets (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Incoming Packets (Sanctuary), ID 58

  2015-11-25 17:18:36,391 INFO  (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.SimianGroupsServicesConnectorModule [SIMIAN-GROUPS-CONNECTOR]: Closing SimianGroupsServicesConnector

  2015-11-25 17:18:36,392 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.XmlRpcGroupsServicesConnectorModule [XMLRPC-GROUPS-CONNECTOR]: Closing XmlRpcGroupsServicesConnector

  2015-11-25 17:18:36,798 INFO  (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: data service http://search.osgrid.org/register.php notified. Secret: 38384e09-f875-4b31-a5a0-dccc4566ac1d

  2015-11-25 17:18:36,801 INFO  (1) - OpenSim.Framework.Servers.BaseOpenSimServer [SHUTDOWN]: Shutdown processing on main thread complete.  Exiting...

   


------------------------------------------------------------------------------


       This email has been checked for viruses by Avast antivirus software. 
        www.avast.com 
       







_______________________________________________Opensim-users mailing listOpensim-users at opensimulator.orghttp://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users 



--------------------------------------------------------------------------------
_______________________________________________
Opensim-users mailing list
Opensim-users at opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-users/attachments/20151126/058c9c1c/attachment-0001.html>


More information about the Opensim-users mailing list