Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007743opensim[REGION] OpenSim Corepublic2015-11-18 08:092018-07-13 19:18
ReporterShy Robbiani 
Assigned ToShy Robbiani 
PrioritynormalSeverityminorReproducibilityalways
StatusacknowledgedResolutionno change required 
PlatformvServer 64-bit/4 GB RAM on i7OSDebian Linux 8 LennieOS Version3.16.0-4-amd64
Product Versionmaster (dev code) 
Target VersionFixed in Version 
Summary0007743: Simulator hangs forever in shutdown on Linux
DescriptionAfter the message "16:03:03 - [SHUTDOWN]: Shutdown processing on main thread complete. Exiting..." or killing the simulator with <ctrl>-C the simulator hangs forever.

I could not restart the simulator before manually killing the process since a port was still in use.

The problem exists in both, grid mode connected to OSGrid and standalone.
I cannot reproduce this behavior in Windows/.NET.
Steps To Reproduce1) Run the simulator
2) enter shutdown, quit or <ctrl>-C from the simulator console
Additional InformationLater this week I will give it a try in another Linux environment as OpenSim in this VM seems to be very unstable. I had several OpenSim crashes today. It's the first time I'm running OpenSim on this box so I'm not 100% sure whether it's my setup (latest stable Debian and Mono) or OpenSim that causes troubles.

Linux Kernel: SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux
Mono Version: Stable 4.2.1.102/6dd2d0d

On another machine with exactly the same Debian release I've seen other network applications not terminating properly.
TagsNo tags attached.
Git Revision or version number0.8.3.0/0fbb4f0
Run ModeStandalone (1 Region) , Grid (1 Region per Sim)
Physics EngineBulletSim
EnvironmentMono / Linux64
Mono VersionOther
ViewerN/A
Attached Fileslog file icon OpenSim.log [^] (32,980 bytes) 2017-04-08 10:20

- Relationships

-  Notes
(0029563)
OtakuMegane (reporter)
2015-11-18 18:19

Confirming the issue here. Stopping the instance with ctrl-c makes it hang and I then have to use kill. Running on an OpenVZ container but Opensim is normally quite stable for me.

CentOS 6
Mono 4.2.1.102
Opensim master/9c5acb3df4173259c414221590761d27fa3af0ed
(0029567)
mewtwo0641 (reporter)
2015-11-19 05:51
edited on: 2015-11-19 05:51

I can't confirm a freeze on Windows .NET requiring manual termination of the process as observed in a Linux environment but I do see that whenever I attempt to shutdown OpenSim, once it reaches the "[SHUTDOWN]: Shutdown processing on main thread complete." point, it freezes for a few seconds, turns white, and eventually I get an error from Windows saying "OpenSim has stopped working" then it gives me the options to check for a solution online or close the program.

There's no stack trace that I can see either in the console window or in the OpenSim log file.

This is on Windows 7 x64 with .NET 4.0

(0029568)
Jak Daniels (reporter)
2015-11-19 05:59

I can confirm this happens to me too on linux. I have to kill -9 the mono process after 'shutdown'.
I do also see one or more 'Timeout detected for thread' messages during the shutdown phase.

Centos 6
Mono 4.2.1.102
Opensim Master @ commit 0fbb4f0067dd6ebda28ba27686686c09de473f83
(0029570)
UbitUmarov (administrator)
2015-11-19 07:49

My development environment is on win 7, and I also can't repo this problem :(
(0029572)
zadark (reporter)
2015-11-19 08:16
edited on: 2015-11-19 08:23

Not seeing this issue.
We have a number of OpenSim Dev/Master instances running on Linux, standalone and grid (including robust) both Bullet and uODE physics. Linux kernels 3.x and 4.x. Mono 4+

Initial issues when updating using git pull resulted in a bin directory with conflicting libraries. We did not investigate this issue.

Fresh clones resolved these issues and various configurations (ini) migrated from the original instances.

We also run Windows instances on Server 2012, similar issues with git update, fresh clones resolved.

When building OpenSim by default we always include rebuild (Linux mono #rebuild /t:rebuild).

(0029573)
Jak Daniels (reporter)
2015-11-19 08:42
edited on: 2015-11-19 09:13

I also started with a fresh git clone of master.

I also tried with and without any addon-modules like OpenSimSearch and OpenSimProfile

I built with:
./runprebuild.sh autoclean && ./runprebuild.sh vs2010 && xbuild /p:Configuration=Release;

OS is Centos 6 (kernel 2.6.32-573.8.1.el6.x86_64)
Mono is version Stable 4.2.1.102/6dd2d0d

Using default configs from master, but with the usual replacements from OSGrid.

After the shutdown sequence reaches
[SHUTDOWN]: Shutdown processing on main thread complete. Exiting...

Opensim hangs, and ps -eLf shows 56 threads still running.
Here is a list of their names if that helps:

ps H -C mono -o 'pid tid cmd comm'
  PID TID CMD COMMAND
17137 17137 mono OpenSim.exe -inimaster mono
17137 17138 mono OpenSim.exe -inimaster mono
17137 17139 mono OpenSim.exe -inimaster Finalizer
17137 17140 mono OpenSim.exe -inimaster STP:Util:0
17137 17141 mono OpenSim.exe -inimaster STP:Util:1
17137 17142 mono OpenSim.exe -inimaster STP:Util:2
17137 17143 mono OpenSim.exe -inimaster STP:Util:3
17137 17144 mono OpenSim.exe -inimaster STP:Util:4
17137 17145 mono OpenSim.exe -inimaster STP:Util:5
17137 17146 mono OpenSim.exe -inimaster STP:Util:6
17137 17147 mono OpenSim.exe -inimaster STP:Util:7
17137 17148 mono OpenSim.exe -inimaster STP:Util:8
17137 17149 mono OpenSim.exe -inimaster STP:Util:9
17137 17150 mono OpenSim.exe -inimaster STP:Util:10
17137 17151 mono OpenSim.exe -inimaster STP:Util:11
17137 17152 mono OpenSim.exe -inimaster STP:Util:12
17137 17153 mono OpenSim.exe -inimaster STP:Util:13
17137 17154 mono OpenSim.exe -inimaster STP:Util:14
17137 17155 mono OpenSim.exe -inimaster STP:Util:15
17137 17156 mono OpenSim.exe -inimaster STP:Util:16
17137 17157 mono OpenSim.exe -inimaster STP:Util:17
17137 17158 mono OpenSim.exe -inimaster STP:Util:18
17137 17159 mono OpenSim.exe -inimaster STP:Util:19
17137 17160 mono OpenSim.exe -inimaster STP:Util:20
17137 17161 mono OpenSim.exe -inimaster STP:Util:21
17137 17162 mono OpenSim.exe -inimaster STP:Util:22
17137 17163 mono OpenSim.exe -inimaster STP:Util:23
17137 17164 mono OpenSim.exe -inimaster STP:Util:24
17137 17165 mono OpenSim.exe -inimaster STP:Util:25
17137 17166 mono OpenSim.exe -inimaster STP:Util:26
17137 17167 mono OpenSim.exe -inimaster STP:Util:27
17137 17168 mono OpenSim.exe -inimaster STP:Util:28
17137 17169 mono OpenSim.exe -inimaster STP:Util:29
17137 17170 mono OpenSim.exe -inimaster STP:Util:30
17137 17171 mono OpenSim.exe -inimaster STP:Util:31
17137 17172 mono OpenSim.exe -inimaster Timer-Scheduler
17137 17179 mono OpenSim.exe -inimaster Non-blocking no
17137 17180 mono OpenSim.exe -inimaster STP:PoolService
17137 17181 mono OpenSim.exe -inimaster PollServiceWork
17137 17182 mono OpenSim.exe -inimaster PollServiceWork
17137 17183 mono OpenSim.exe -inimaster PollServiceWork
17137 17184 mono OpenSim.exe -inimaster PollServiceWatc
17137 17185 mono OpenSim.exe -inimaster STP:ScriptsHttp
17137 17186 mono OpenSim.exe -inimaster mono
17137 17187 mono OpenSim.exe -inimaster mono
17137 17188 mono OpenSim.exe -inimaster mono
17137 17189 mono OpenSim.exe -inimaster mono
17137 17194 mono OpenSim.exe -inimaster MapItemRequestT
17137 17195 mono OpenSim.exe -inimaster MapBlockSendThr
17137 17201 mono OpenSim.exe -inimaster MeshWorkerThrea
17137 17202 mono OpenSim.exe -inimaster MeshWorkerThrea
17137 17203 mono OpenSim.exe -inimaster TextureWorkerTh
17137 17204 mono OpenSim.exe -inimaster TextureWorkerTh
17137 17210 mono OpenSim.exe -inimaster STP:XEngine:0
17137 17211 mono OpenSim.exe -inimaster STP:XEngine:1

(0029575)
UbitUmarov (administrator)
2015-11-19 16:58

most of those threads are service ones that should be in wait state, and should die with no problems
but at least one is stuck somewhere...
(0029593)
Shy Robbiani (reporter)
2015-11-21 05:35

It seems to be a problem with Mono 4.2. After I replaced Mono Stable 4.2.1.102/6dd2d0d with Mono Stable 4.0.5.1/1d8d582 the problem disappeared.
(0029621)
mewtwo0641 (reporter)
2015-11-23 20:07

I'm not sure if this is related or not but I happened to come across this exception upon my latest code pull and shutdown:

2015-11-23 22:04:18,488 ERROR (1) - OpenSim.OpenSimBase [SHUTDOWN]: Ignoring failure during shutdown -
System.MissingMethodException: Method not found: 'Boolean OpenSim.Framework.Servers.HttpServer.BaseHttpServer.RemoveAgentHandler(System.String, OpenSim.Framework.Servers.HttpServer.IHttpAgentHandler)'.
   at OpenSim.ApplicationPlugins.Rest.RestPlugin.RemoveAgentHandler(String agentName, IHttpAgentHandler handler)
   at OpenSim.ApplicationPlugins.Rest.Inventory.RestHandler.Close() in c:\Users\Admin\Desktop\ostest\opensim\OpenSim\ApplicationPlugins\Rest\Inventory\RestHandler.cs:line 347
   at OpenSim.ApplicationPlugins.Rest.RestPlugin.Dispose() in c:\Users\Admin\Desktop\ostest\opensim\OpenSim\ApplicationPlugins\Rest\RestPlugin.cs:line 370
   at OpenSim.OpenSimBase.ShutdownSpecific() in c:\Users\Admin\Desktop\ostest\opensim\OpenSim\Region\Application\OpenSimBase.cs:line 894
(0029638)
Shy Robbiani (reporter)
2015-11-24 09:28

I closed this as it definitely seems to be an issue with Mono Stable 4.2.1.102. After I switched to Mono Stable 4.0.5.1/1d8d582 the problem disappeared.

Just one more note: after I switched to Mono Stable 4.0.5.1/1d8d582 on Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux the problem disappeared but I experiences irregular crashes.

After stepping back to Mono 3.2.8 (Debian 3.2.8+dfsg-10) as it's included with the Debian distribution the system runs stable and all shutdowns terminate as expected.
(0029684)
slow putzo (reporter)
2015-11-26 07:58

I wanted to update this mantis to say I also have this same issue running Fedore distro.

Release Notes

You are at 2573869.9, 2556494.5, 22.2 in Sanctuary located at pool-100-13-12-235.tampfl.fios.verizon.net (100.13.12.235:9137)
OpenSim 0.9.0.0 Dev OSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23 (Unix/Mono)
Mono JIT compiler version 4.2.1 (Stable 4.2.1.102/6dd2d0d Thu Nov 12 04:43:41 EST 2015)

The simulators appear to be running fine, but kind of a nuisance not to be able to use normal commands to shut them down.

 

They are running in a “screen” terminal window.

 

Here is the log of a shutdown immediately after launching a region.

 

2015-11-25 17:18:29,952 INFO (1) - OpenSim.Framework.Servers.ServerBase [SERVER BASE]: Running shutdown_commands.txt

2015-11-25 17:18:29,978 INFO (1) - OpenSim.Framework.Servers.ServerBase [SERVER BASE]: Running 'backup'

2015-11-25 17:18:33,925 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing all threads

2015-11-25 17:18:33,926 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing listener thread

2015-11-25 17:18:33,926 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing clients

2015-11-25 17:18:33,926 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing console and terminating

2015-11-25 17:18:33,927 INFO (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Closing down the single simulator: Sanctuary

2015-11-25 17:18:34,148 DEBUG (Maintenance (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Maintenance (Sanctuary), ID 79

2015-11-25 17:18:34,428 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: TriggerSceneShuttingDown

2015-11-25 17:18:34,428 INFO (1) - OpenSim.Region.Framework.Scenes.EventManager [EVENT MANAGER]: TriggerSceneShuttingDown done

2015-11-25 17:18:34,429 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Persisting changed objects

2015-11-25 17:18:34,588 DEBUG (1) - OpenSim.Services.GridService.GridService [GRID SERVICE]: Deregistering region Sanctuary (3b15c050-1fe4-11df-8a39-0800200c9a66) at 10054-9986

2015-11-25 17:18:34,909 DEBUG (1) - OpenSim.OpenSimBase [SHUTDOWN]: Shutting down region Sanctuary

2015-11-25 17:18:35,678 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice]: region Sanctuary: uuid 3b15c050-1fe4-11df-8a39-0800200c9a66: located directory id 4439471

2015-11-25 17:18:35,773 DEBUG (Threadpool worker) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Heartbeat-(Sanctuary), ID 75

2015-11-25 17:18:35,774 ERROR (Threadpool worker) - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat-(Sanctuary)". ThreadState=Stopped. Last tick was 1916ms ago.

2015-11-25 17:18:36,025 INFO (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: Region Sanctuary is being removed, removing from indexing

2015-11-25 17:18:36,031 INFO (1) - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Shutting down 487 scripts in Sanctuary

2015-11-25 17:18:36,208 INFO (1) - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Shutting down the LLUDP server for Sanctuary

2015-11-25 17:18:36,208 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping outbound UDP loop

2015-11-25 17:18:36,209 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping inbound UDP loop

2015-11-25 17:18:36,210 DEBUG (Outgoing Packets (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Outgoing Packets (Sanctuary), ID 59

2015-11-25 17:18:36,223 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread0, ID 66

2015-11-25 17:18:36,223 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread1, ID 67

2015-11-25 17:18:36,231 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Dispose Physics

2015-11-25 17:18:36,234 DEBUG (bulletunmanaged (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread bulletunmanaged (Sanctuary), ID 57

2015-11-25 17:18:36,246 DEBUG (Incoming Packets (Sanctuary)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Incoming Packets (Sanctuary), ID 58

2015-11-25 17:18:36,391 INFO (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.SimianGroupsServicesConnectorModule [SIMIAN-GROUPS-CONNECTOR]: Closing SimianGroupsServicesConnector

2015-11-25 17:18:36,392 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.XmlRpcGroupsServicesConnectorModule [XMLRPC-GROUPS-CONNECTOR]: Closing XmlRpcGroupsServicesConnector

2015-11-25 17:18:36,798 INFO (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: data service http://search.osgrid.org/register.php [^] notified. Secret: 38384e09-f875-4b31-a5a0-dccc4566ac1d

2015-11-25 17:18:36,801 INFO (1) - OpenSim.Framework.Servers.BaseOpenSimServer [SHUTDOWN]: Shutdown processing on main thread complete. Exiting...

 
I'm certainly not saying it is not a mono issue, but I have been on release 4.x.x ever since OSgrid came back online with no issues whatever.

OSgrid release of opensim OSgrid 0.8.3.0 (Dev) 41b2855: 2015-10-21
Runs without any issues on this version of mono however
OSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23 fails with this issue

Obviously something has changed in opensim, and if that change is a legal change and does not violate some mono restriction or specification they yes mono is broken.

A developer of opensim who know "what" is broken should report this to mono so it can be fixed, and I hope they are doing that.

As a side note, the reason I moved up to newer releases of mono from the Fedora distros 2.8 version was at the suggestion of Nebandon who was running mono 4.0.x at the time and was not experiencing issues I was with the old 2.8 release of mono.

I have not detected any other issues other than is shutdown issue, so I am going to remain at mono 4.2.x and just use "kill" for now.

I think dropping back to a much older release of mono is a very poor resolution to a problem that was not present one month ago in opensim without a detailed explanation that it is indeed a mono issue and not just a "guess" that it is.





(0029699)
slow putzo (reporter)
2015-11-26 13:05

More information on this bug.

To say windows with .net is working correctly is also false. It is indeed throwing an ERROR at shutdown and could well be what is causing mono to hang.

Here is a shutdown in windows using .net for the second to the last version of opensim for OSgrid, and the last version of opensim released for OSgrid.

You can clearly see there are no ERRORS in shutdown previously, and there is after the current release.

There is something wrong that was not wrong before in mono and in .net. The difference is that .NET still closes, and mono does not.

OSgrid 0.8.3.0 (Dev) 41b2855: 2015-10-21

2015-11-26 15:46:32,310 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing all threads
2015-11-26 15:46:32,310 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing listener thread
2015-11-26 15:46:32,310 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing clients
2015-11-26 15:46:32,310 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing console and terminating
2015-11-26 15:46:32,310 INFO (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Closing down the single simulator: Development
2015-11-26 15:46:32,357 DEBUG (Heartbeat-(Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Heartbeat-(Development), ID 67
2015-11-26 15:46:32,420 DEBUG (Maintenance (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Maintenance (Development), ID 72
2015-11-26 15:46:32,826 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Persisting changed objects
2015-11-26 15:46:37,029 DEBUG (1) - OpenSim.Services.GridService.GridService [GRID SERVICE]: Deregistering region Development (570c1600-3516-11e4-8c21-0800200c9a66) at 10054-9986
2015-11-26 15:46:37,295 DEBUG (1) - OpenSim.OpenSimBase [SHUTDOWN]: Shutting down region Development
2015-11-26 15:46:37,310 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_chan_search.php?cond_channame=570c1600-3516-11e4-8c21-0800200c9a66D&auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235> [^]
2015-11-26 15:46:37,466 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice]: region Development: uuid 570c1600-3516-11e4-8c21-0800200c9a66: located directory id 4443169
2015-11-26 15:46:37,482 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_chan_search.php?&cond_chanparent=4443169&auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235> [^]
2015-11-26 15:46:37,576 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_chan_mod.php?mode=delete&chan_id=4443169&auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235> [^]
2015-11-26 15:46:37,717 INFO (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: Region Development is being removed, removing from indexing
2015-11-26 15:46:37,748 INFO (1) - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Shutting down 492 scripts in Development
2015-11-26 15:46:37,842 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread0, ID 58
2015-11-26 15:46:37,857 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread1, ID 59
2015-11-26 15:46:37,857 INFO (1) - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Shutting down the LLUDP server for Development
2015-11-26 15:46:37,857 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping outbound UDP loop
2015-11-26 15:46:37,857 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping inbound UDP loop
2015-11-26 15:46:37,904 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice] Sending request <http://www.osp.vivox.com/api2/viv_signout.php?auth_token=thomgate3149-admin:1498570718:291df246153ec7c5bdbaab3fb3dbcba1:100.13.12.235> [^]
2015-11-26 15:46:37,967 DEBUG (Outgoing Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Outgoing Packets (Development), ID 62
2015-11-26 15:46:37,967 DEBUG (Incoming Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Incoming Packets (Development), ID 61
2015-11-26 15:46:37,982 DEBUG (bulletunmanaged (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread bulletunmanaged (Development), ID 57
2015-11-26 15:46:37,998 INFO (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.SimianGroupsServicesConnectorModule [SIMIAN-GROUPS-CONNECTOR]: Closing SimianGroupsServicesConnector
2015-11-26 15:46:37,998 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.XmlRpcGroupsServicesConnectorModule [XMLRPC-GROUPS-CONNECTOR]: Closing XmlRpcGroupsServicesConnector
2015-11-26 15:46:38,092 INFO (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: data service http://search.osgrid.org/register.php [^] notified. Secret: 24383411-9692-40df-b8f1-e22e2bb9aa45
2015-11-26 15:46:38,107 INFO (1) - OpenSim.Framework.Servers.BaseOpenSimServer [SHUTDOWN]: Shutdown processing on main thread complete. Exiting...


OSgrid 0.9.0.0 (Dev) 5e4b166: 2015-11-23

2015-11-26 15:49:02,654 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing all threads
2015-11-26 15:49:02,654 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing listener thread
2015-11-26 15:49:02,654 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Killing clients
2015-11-26 15:49:02,654 INFO (1) - OpenSim.OpenSimBase [SHUTDOWN]: Closing console and terminating
2015-11-26 15:49:02,654 INFO (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Closing down the single simulator: Development
2015-11-26 15:49:03,170 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: TriggerSceneShuttingDown
2015-11-26 15:49:03,170 INFO (1) - OpenSim.Region.Framework.Scenes.EventManager [EVENT MANAGER]: TriggerSceneShuttingDown done
2015-11-26 15:49:03,170 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Persisting changed objects
2015-11-26 15:49:03,186 DEBUG (Maintenance (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Maintenance (Development), ID 77
2015-11-26 15:49:03,217 DEBUG (36) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Heartbeat-(Development), ID 72
2015-11-26 15:49:03,217 ERROR (36) - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat-(Development)". ThreadState=Stopped. Last tick was 625ms ago.
2015-11-26 15:49:03,295 DEBUG (TouchAllSceneAssets) - OpenSim.Region.CoreModules.Asset.FlotsamAssetCache [FLOTSAM ASSET CACHE]: Could not find asset ce5301d9-0943-4ffa-891e-f8305b748897, type 0 referenced by object @-Heather Genie Dressing Booth at <48.14737, 75.16174, 22.52853> in scene Development when pre-caching all scene assets
2015-11-26 15:49:23,795 DEBUG (1) - OpenSim.Services.GridService.GridService [GRID SERVICE]: Deregistering region Development (570c1600-3516-11e4-8c21-0800200c9a66) at 10054-9986
2015-11-26 15:49:24,061 DEBUG (1) - OpenSim.OpenSimBase [SHUTDOWN]: Shutting down region Development
2015-11-26 15:49:24,280 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.Voice.VivoxVoice.VivoxVoiceModule [VivoxVoice]: region Development: uuid 570c1600-3516-11e4-8c21-0800200c9a66: located directory id 4443177
2015-11-26 15:49:24,686 INFO (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: Region Development is being removed, removing from indexing
2015-11-26 15:49:24,733 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread0, ID 64
2015-11-26 15:49:24,749 DEBUG (1) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread InventoryWorkerThread1, ID 65
2015-11-26 15:49:24,749 INFO (1) - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine]: Shutting down 492 scripts in Development
2015-11-26 15:49:24,827 INFO (1) - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Shutting down the LLUDP server for Development
2015-11-26 15:49:24,827 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping outbound UDP loop
2015-11-26 15:49:24,827 DEBUG (1) - OpenMetaverse.OpenSimUDPBase [UDPBASE]: Stopping inbound UDP loop
2015-11-26 15:49:24,842 DEBUG (MapItemRequestThread (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread MapItemRequestThread (Development), ID 58
2015-11-26 15:49:24,858 DEBUG (Outgoing Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Outgoing Packets (Development), ID 68
2015-11-26 15:49:24,858 DEBUG (1) - OpenSim.Region.Framework.Scenes.Scene [SCENE]: Dispose Physics
2015-11-26 15:49:24,889 DEBUG (Incoming Packets (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread Incoming Packets (Development), ID 67
2015-11-26 15:49:24,952 DEBUG (bulletunmanaged (Development)) - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread bulletunmanaged (Development), ID 57
2015-11-26 15:49:25,014 INFO (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.SimianGroupsServicesConnectorModule [SIMIAN-GROUPS-CONNECTOR]: Closing SimianGroupsServicesConnector
2015-11-26 15:49:25,014 DEBUG (1) - OpenSim.Region.OptionalModules.Avatar.XmlRpcGroups.XmlRpcGroupsServicesConnectorModule [XMLRPC-GROUPS-CONNECTOR]: Closing XmlRpcGroupsServicesConnector
2015-11-26 15:49:25,093 INFO (1) - OpenSim.Region.DataSnapshot.DataSnapshotManager [DATASNAPSHOT]: data service http://search.osgrid.org/register.php [^] notified. Secret: d73bfc15-b3db-46b6-800c-d5d320185478
2015-11-26 15:49:25,108 INFO (1) - OpenSim.Framework.Servers.BaseOpenSimServer [SHUTDOWN]: Shutdown processing on main thread complete. Exiting...

As I said, I have no proof this is the mono problem, but there is something wrong and it sure is suspicious that it might well be what is causing mono to hang and not close.
(0029717)
aiaustin (developer)
2015-11-27 01:57
edited on: 2015-11-27 02:34

I also see OpenSim.exe hanging when I "q" out of an idle simulator.. on Windows 10/.NET with latest today's opensim-0.9.0-105-g3029080.zip and on a freshly created standalone, using default SQLite with a single region set up for tests on other issues.

I see one final [WATCHDOG] red timeout message and then the console hangs.. and I left it for 5 minutes before I forcibly terminated it.

09:54:45 - [Scene]: The avatar has left the building
Region (Vue) # q
09:55:01 - [SHUTDOWN]: Closing all threads
09:55:01 - [SHUTDOWN]: Killing listener thread
09:55:01 - [SHUTDOWN]: Killing clients
09:55:01 - [SHUTDOWN]: Closing console and terminating
09:55:01 - [SCENE]: Closing down the single simulator: Vue
09:55:01 - [SCENE]: TriggerSceneShuttingDown
09:55:01 - [EVENT MANAGER]: TriggerSceneShuttingDown done
09:55:01 - [SCENE]: Persisting changed objects
09:55:01 - [WATCHDOG]: Removing thread Maintenance (Vue), ID 37
09:55:03 - [WATCHDOG]: Removing thread Heartbeat-(Vue), ID 34
09:55:03 - [WATCHDOG]: Timeout detected for thread "Heartbeat-(Vue)". ThreadState=Stopped. Last tick was 2438ms ago.

----
Note added: I am now suspecting this was because the SQLite data base was still persisting the objects loaded in a previous OAR, as on SQLite this can take a long time. It was just not clear whether or not the console had hung, there is no warning that objects are not yet fully persisted***, and the watchdog timeout red error as the last thing showing led me to terminate the process after 5 minutes. However, when I restarted many objects that should have been in the region were missing.

After a load oar, I don't think an immediate backup and persist of the objects is done or confirmed to the console. It is necessary to manually do a backup on the console or wait for the objects to be persisted. This can take a long time on SQLite if a big load oar has been done. When I retested this the backup took over 15 minutes.. hence the hang on "q" initially.

*** maybe at some stage a suggestion for improvement in this area ought to be made as it can lead to serious loss of content and data base corruption.

(0029720)
UbitUmarov (administrator)
2015-11-27 07:33

removed some of the timeouts messages with ThreadState=Stopped, they where just missing notification of thread termination to watchdog, so no impact on the issue, so this was just cleanup :(
09:55:01 - [SCENE]: Persisting changed objects - is a synchronous operation.
(0029721)
aiaustin (developer)
2015-11-27 08:01

I see quite a few extra debug messages generally Ubit.. some might just be temporary, but they are very volumous.. should they not be only coming out if debug levels are set higher than the default? E.g...

11:54:24 - [CompleteMovement]: WaitForUpdateAgent: 15ms
11:54:24 - [MakeRootAgent]: enter lock: 0ms
11:54:24 - [MakeRootAgent]: out lock: 0ms
11:54:24 - [MakeRootAgent]: Grouptitle: 16ms
11:54:24 - [MakeRootAgent]: TriggerSetRootAgentScene: 16ms
11:54:24 - [MakeRootAgent]: position and physical: 16ms
11:54:24 - [MakeRootAgent]: TriggerOnMakeRootAgent and done: 94ms
11:54:24 - [CompleteMovement]: MakeRootAgent: 125ms
11:54:24 - [CompleteMovement]: MoveAgentIntoRegion: 125ms
11:54:24 - [CompleteMovement]: ReleaseAgent: 125ms
11:54:24 - [CompleteMovement]: ValidateAndSendAppearanceAndAgentData: 140ms
11:54:24 - [CompleteMovement]: attachments: 140ms
11:54:24 - [CompleteMovement]: openChildAgents: 156ms
11:54:24 - [CompleteMovement]: SendInitialDataToMe: 156ms
11:54:24 - [CompleteMovement]: end: 156ms
(0029725)
slow putzo (reporter)
2015-11-27 12:50

After a long day of walking the commit path here is what I discovered.

My impression is any commit that starts with opensim-0.6.9.rc1- fails to shutdown, or has compile errors which I did not even try to test.

The only test I did was try to do a shutdown.

It looks like the opensim-0.6.9.rc1- release had this bug in it from the very start. When another developer did a commit that did not start with opensim-0.6.9.rc1- that commit would compile fine, and would shutdown fine.

I thought opensim was at release 0.8.3 prior to going to 0.9.0
Then I have no idea how the commit stuff works as I have no knowledge in that area at all. I could not even find this old release because my archive only goes back to 2011 and there the release was osgrid.opensim-04072011.v0.7.1.abea0c7

Unfortunately it looks like release opensim-0.6.9.rc1- became release 0.9.0 so now that has this bug in it as well.

opensim-0.9.0-102-g9afe2b0 SHUTDOWN FAILS

2015-10-21 23:47 Diva Canto Fix an issue introduced in 70a46fe0907c822a5244e36 SHUTDOWN FAILS - opensim-0.6.9.rc1-15867
2015-10-21 23:47 Diva Canto Fix an issue introduced in 70a46fe0907c822a5244e36 SHUTS DOWN NORMALLY - 41B2855
2015-10-21 00:11 UbitUmarov fix or remove some wrong ODE configuration settin SHUTDOWN FAILS - opensim-0.6.9.rc1-15861
2015-10-20 14:37 UbitUmarov STATUS_ROTATE are linkset flags and not prim SHUTDOWN FAILS - opensim-0.6.9.rc1-15851
2015-10-20 01:12 UbitUmarov update ODE windows DLL libraries to a modified ver WILL NOT COMPILE - opensim-0.6.9.rc1-15850
2015-10-19 22:58 Melanie Thielker Let the initiator of a teleport or crossing know t SHUTS DOWN NORMALLY - 2B437F8
2015-09-06 20:46 UbitUmarov add lost admin_reset_land method SHUTDOWN FAILS - opensim-0.6.9.rc1-15616
2015-09-03 19:51 UbitUmarov fix a old bug i added: MaxWearables can't be chang SHUTDOWN FAILS - opensim-0.6.9.rc1-15596-g7bfa311
2015-09-03 19:22 UbitUmarov fix CM_api compile error COMPILE ERRORS - opensim-0.6.9.rc1- 15595
2015-09-03 17:50 UbitUmarov missing file COMPILE ERRORS - opensim-0.6.9.rc1-15593-gb265e35.zip
2015-09-03 17:39 UbitUmarov at last we can login and see objects ( friends is COMPILE ERRORS - opensim-0.6.9.rc1-15592
2015-09-02 18:54 UbitUmarov seems to compile ( tests comented out) COMPILE ERRORS - opensim-0.6.9.rc1-15591
2015-09-01 16:20 Diva Canto Moved ExtendedPhysics from OptionalModules to Bull SHUTS DOWN NORMALLY - 0235E58
2015-09-01 13:54 UbitUmarov bad merge? COMPILE ERRORS - opensim-0.6.9.rc1-15590
2015-09-01 10:48 UbitUmarov remove lixo SHUTS DOWN NORMALLY - F811FDF
2015-09-01 10:43 UbitUmarov Merge remote-tracking branch 'os/master' SHUTS DOWN NORMALLY - FB78B18
2015-09-01 10:40 UbitUmarov lixo NOTHING IN THIS, JUST ONE FILE - 52F7991
2015-08-23 17:25 UbitUmarov fix db region find by range for varregions ( igno COMPILE ERRORS - opensim-0.6.9.rc1-13940
2015-08-18 22:59 UbitUmarov change pollService stop() to send 503 error and no COMPILE ERRORS - opensim-0.6.9.rc1-13898
2015-08-18 20:32 UbitUmarov do keepalive on mesh and texture GET. Dont use reu COMPILE ERRORS - opensim-0.6.9.rc1-13897
2015-08-18 20:03 UbitUmarov try to serialize http requests from same connectio COMPILE ERRORS, DOES NOT RUN - opensim-0.6.9.rc1-13896

Today I had to revert my regions back to the OSgrid 2015-10-23 version because my residents were driving me nuts with things that did not work and I could not deal with all the problems. Once I did the revert, the complaining stopped.
(0029726)
aiaustin (developer)
2015-11-27 14:09
edited on: 2015-11-28 09:36

slow putzo... 0.8.3 was in use briefly for dev master after the 0.8.2 release was branched off. Some development of 0.8.3 dev master continued and those have opensim-####### (using the first characters of the full long Git commit code) style naming for viewgit downloads as used previously.

The 0.6.9.rc1 download moniker was a viewgit file naming issue from the avination code merge commits that had been developed over the last month or more on a separate branch. That viewgit download file naming issue was resolved by @Melanie, though all commits in place remain with the faulty file naming for downloads. Due to the large number of changes introduced by the avination code merge the devs chose then to switch the viewgit label to 0.9.0-nnn-g####### (nnn being a sequence number, note the "g" for git and then the first 7 hex characters of the commit code).

The actual code version change to report in code 0.9.0.0 was done by @Diva and followed a few days later, so some viewgit downloads labelled as 0.9.0-nnn-g####### report their version number as 0.8.3.0.

Since the avination code was worked on over the last month or more in parallel with the main dev master, there are commits done on the original code base that overlap in time with commits now merged in from the avination branch... so trying to separate out now the commits on a timeline basis will be very hard. When all the avination code was merged the commits have dates that interleave those original main branch commits. That's why you see some commits labelled for download as just opensim-#######.zip in between those with the faulty 0.6.9.rc1 and more recently the 0.9.0 file formats.

I hope that helps a bit in unravelling where the faults were introduced.

(0029728)
slow putzo (reporter)
2015-11-27 15:03

That does explain why some commits do work interspersed with those that don't. It would appear this problem was in the very first commit I could find that compiled without errors and was labeled opensim-0.6.9.rc1-xxxxx.
The very first commits with than faulty name would not compile clean for me, and in fact required me to over-ride the error message windows was giving saying the program was not "valid" or something like that. Most of those back in early August were like that.

I hope this helps in some small way. There isn't much else I know how to do in terms of helping find what this issue is.

I am keeping my home region at the current release levels, as "I" do not notice any other issues other than the shutdown problem which I can work around with the kill command.
On the positive side, I do have much better "tp" experience and border crossings are much smoother.

2015-09-03 19:51 UbitUmarov fix a old bug i added: MaxWearables can't be chang SHUTDOWN FAILS - opensim-0.6.9.rc1-15596-g7bfa311

Was the earliest commit I could find that didn't have compile errors and it immediately followed a commit that did have compile errors.

Everything before the code merge started worked fine for shutdown.

I know that isn't much help. I don't think this is something that was working and then got broke, but rather existed in the code that was being merged.
I only use the code releases OSgrid puts out for my region updates, and it is very seldom I ever use a git release unless it fixes something I was having a specific issue with.
(0029729)
UbitUmarov (administrator)
2015-11-27 15:21

It is clear to me that issue is on the merged branch.
Can be code imported from avination, where the issue never happened, like it doesn't happen to me and others,
or it can be a bad git merge effect lost somewhere..
or even something I did messed up on this process...
(0029730)
slow putzo (reporter)
2015-11-27 15:28

If there is anything I can do to assist in finding what the problem is, I will be happy to help. I'm not a programmer, but have a little experience now with at least testing to see if it fails or not. The problem is absolutely repeatable for me every time.
(0029732)
slow putzo (reporter)
2015-11-27 16:23

UbitUmarov, I just compiled the latest commit you made after several of your commits today and it now shuts down correctly for me.
This is the latest commit that I just compiled and tested:
 2015-11-27 23:46 UbitUmarov remove terrain height clamping left over the ushor master SHUTS DOWN NORMALLY - opensim-0.9.0-118-g9928076.zip
The previous one I tested was:
2015-11-26 16:29 Melanie Thielker Mantis 0007765: Add new ClampNegativeZ option. Defau SHUTDOWN FAILS opensim-0.9.0-102-g9afe2b0.zip
I am logged in on my region and so far everything appears as it should be.

If you want me to walk through the several commits to see which one fixed it, I will be happy to do that and report it back here. Maybe you already know what fixed it.
(0029733)
UbitUmarov (administrator)
2015-11-27 16:28

this can also be random :(
but I did a few changes on some threads control.
(0029734)
OtakuMegane (reporter)
2015-11-27 16:39

Just tried the latest commit as well and both shutdown command and ctrl-c are working normally with it.
(0029735)
slow putzo (reporter)
2015-11-27 16:47

ok, I will find when it started to work for sure, so you will know if it becomes random for me or if it indeed is fixed. I did just find another failure and confirmed that it does not work in V0.9.0 but it has nothing to do with this problem.
(0029736)
UbitUmarov (administrator)
2015-11-27 16:49

thx
and let us know that other failure
(0029737)
OtakuMegane (reporter)
2015-11-27 16:59

Poked around some and found "f16d36627f949a3568fea09dddcb76c9f8d823e6 - change threading on GetTexture and GetMesh NonSharable region modules" is where the problem got fixed.
(0029738)
UbitUmarov (administrator)
2015-11-27 17:00

thx :)
(0029739)
slow putzo (reporter)
2015-11-27 17:01

see:
http://opensimulator.org/mantis/view.php?id=7768 [^]
For the other problem
(0029744)
aiaustin (developer)
2015-11-28 01:21
edited on: 2015-11-28 01:24

Just a note for Ubit and others, as I see some work going on in the area of FetchInventoryDesc and pending event handling. iN the past we have had issues with some pending inventory fetch events never getting handled, or timing out after a very long time even after logoff, especially on OSGrid. This is an area we have had major grief with over the years. In one case inventory fetch hangs took nearly two years to pin down in work between JustinCC, Mata, myself and others. It was a totally major thing for some people with more complex setups. It got fixed, but it was one of those fine timing things. Please don't mess with thread timing, pending event completion assumptions, or thread count expansion without thinking if its necessary, and then thinking again, and again,.. before a change is made. Ai (grief avoidance officer) :-)

(0029802)
slow putzo (reporter)
2015-12-11 16:49

This problem has returned in the OSgrid 0.9.0.0 (Dev) 7d8b783: 2015-12-05 release of opensim distributed by OSgrid.

The failure is identical. I had stopped using git versions because the one problem I am having with a very complex script is so difficult to try and explain I decoded to wait for the OSgrid releases and see if they fixed that problem.

This shutdown problem does not happen on my windows instance, but it does give the "red timeout error" when shutting down.

I do not know if the two are actually linked.

2015-12-11 19:17:27,448 ERROR (82) - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "MapBlockSendThread (Development)". ThreadState=Stopped. Last tick was 3011000ms ago.
(0029835)
OtakuMegane (reporter)
2015-12-19 15:19

I still compile from git, have never actually run the OSGrid download itself. Currently on version e095f51b05f980474cf8a43594025a46ee6fa0cf and I have not encountered the problem yet. I went back and compiled for the 12-5-15 OSGrid version and did not have the issue return there either.
(0029847)
slow putzo (reporter)
2015-12-21 19:54

I just ran the latest GIT (opensim-0.9.0-216-g6437a94) as of today, and ran it on my test region. Using both physics engines the shutdown worked just fine.

There is a rather big difference in how fast the region comes online between the two physics engines.

 This is a nearly empty region with only 4 objects on it. All of these objects are currently undergoing mantis scripting bug fixes.
(0029849)
UbitUmarov (administrator)
2015-12-22 03:20

with only 4 objects there should not be a big difference..
But both Ode and ubOde now may seem to delay the startup.
Reason is that now there is a initialization step where objects physics model is built. So when main loop starts moving objects, they are really ready for it.
Some large meshs (that should be avoid on physics) can take sometime to build the physics model
You can see that step in the log just before start of heartbeat
The shutdown issue is strange, because there where no changes on the code areas we found before being the cause
(0029893)
Robert Adams (administrator)
2015-12-27 21:31

Another data point: if I start up my OSGrid BulletSim regions (on Ubuntu 14.04, Mono JIT 3.2.8), if I immediately try to shut the simulator down with a 'q' on the console, the log gets down to the last SHUTDOWN message but never exits -- it sits forever and a kill is required.

If, on the other hand, I start the region and wait for the hypergrid friend messages to complete, the simulator shuts down with no problem. The hypergrid friend presence posting takes a while (minutes) as some of my friends are from grids that don't exist so there is a long DNS timeout trying to get the grid DNS address failure.

It looks like the friend presence posting threads can keep the simulator from shutting down.
(0031731)
slow putzo (reporter)
2017-04-08 09:49

After holding off doing any updates of my operating system, with Fedora 22 now past end of life, I updated to Fedora 25 by going through an update from release 22, 23, 24, to the current release of 25. This problem returned immediately as I updated from release 22 to release 23. I did not track the versions of mono during the updates until I reached the final goal of getting to release Fedora 25.

I am testing on a different computer than my production regions are running on which are still running Fedora release 22 and do not have this issue.

I am attaching the log file to the initial start of a fresh copy of opensim with a blank database running ubODE and it is a 4 by 4 VAR instance. It is the only thing running on the Fedora 25 machine. I entered the "quit" command and it went to the final shutdown step and hangs. This is exactly what it was doing before.
I am attempting to get the current release of both Linux and mono to report here.

opensim release: OSgrid 0.9.1.0 (Dev) 4499355: 2017-04-01
Linux: Fedora 25 Kernel Linux 4.10.8-200.fc25.x86_64 x86_64
mono: 4.8.0 (Stable 4.8.0520/8f6d0f6 Wed Mar 15 10:51:49 UTC 2017)

I am trying to figure out how to include the log file and it appears I need to finish this note first.
(0031732)
slow putzo (reporter)
2017-04-08 10:06

I was able to attach the opensim.log file to this mantis report.

I notice a very strange thing about the log file.
version: OpenSim 0.9.1.0 Dev OSgrid 0.9.1.0 (Dev) 0170696: 2016-11-26

That date is not what is in the .version file in the "bin" directory. It has this information in it: OSgrid 0.9.1.0 (Dev) 4499355: 2017-04-01

The ZIP file name this release came out of is: osgrid-opensim-04012017.v0.9.1.4499355.zip

I am running the singularity viewer and am able to go to the region and while it is just the bump in the ocean with zero prims and scripts I can fly around just fine and all appears to be ok.

It just will not shut down correctly and you have to use the "kill" command so you can get rid of the open port to restart it.

I hope even though this is a very old mantis this problem can be once again reviewed. Someone just starting to use opensim and using Linux with the current mono is not going to understand why they would need to go back to an ancient release to get opensim to function correctly at shutdown.

My servers that DO shutdown correctly using this same confusing version of opensim put out by OSgrid are running:
Fedora 22
Mono 4.2.3 (Stable 4.2.3.4/832de4b Tue Mar 15 11:39:53 EDT 2016)

I say confusing because the information in the log file does not match what the release is suppose to be or what is in the .version file in the bin directory.
(0031733)
nebadon (administrator)
2017-04-08 10:17

I have this problem on any mono newer than 4.0.4 If you want to install mono 4.0.4 yourself you can use my instructions, if you get stuck just let me know some newer mono has problem with compiling the older mono, but i have a work around for this that I have not documented yet.

https://www.nebadon2025.com/opensim/viewtopic.php?f=7&t=44 [^]
(0031734)
slow putzo (reporter)
2017-04-08 10:29

Please ignore the "confusion" statement. I noticed after posting the log file that it was a log file of several restarts and was not a log of a single fresh restart. I deleted that log file and replaced it with the single "fresh restart" section.

Nebadon, I will try to fall back to version 4.2.3 since it is running fine on my other three servers. I'll post if that does work.

Not sure using old versions of mono is the best idea given they apparently do not believe this problem is a mono issue unless no opensim dev has ever reported it to the mono devs.
(0031735)
nebadon (administrator)
2017-04-08 10:31

I believe at some point it was reported, but generally Mono devs need a example code that repeats the problem reliably, and Opensimulator is far to complicated for them to easily debug the problem with.
(0031736)
nebadon (administrator)
2017-04-08 10:32

Also from my testing there is virtually no performance improvement between mono 4.0.4 and the very latest git master head version of mono, if there is any it is very minimal.
(0031737)
slow putzo (reporter)
2017-04-08 10:43

nebadon,

Since I am able to shutdown a region using the web site interface is there even a reason I care if the quit command does not exit?
I normally do my nightly reboots without doing anything to shutdown the regions, I simply reboot the server and let it bring everything back online at 4 am every night. The only time I have ever needed to do a quit was if I wanted to move the region or make some kind of name change etc. The function in the web interface gives me that ability I believe. As far as OSgrid is concerned the quit probably did complete and the only extra step I need might only be to kill the task in my server on such occasions.
(0031738)
nebadon (administrator)
2017-04-08 10:44

I have regions running on 4.0.4 that have been up for 500+ days without a restart :)
(0031739)
slow putzo (reporter)
2017-04-08 10:52

You are blessed. In Florida I would be happy if we could keep the power on for 500+ days let alone our phones running and Internet access working for even 180+ days at a time.

I think we do a doomsday exercise ever few month here with extended outages of one or more utilities.
(0031745)
tampa (reporter)
2017-04-09 13:34

@nebadon The main reason to upgrade to newer mono is for potential reduction in the obvious memory leak OpenSim has. After 18 days I had a region consume nearly 10GB of memory when inworld osGetPhysicalMemory only reported 2GB of memory used. I have the hope that newer versions of mono will somewhat reduce this problem hence why I upgraded in the first place. The shutdown problem is a bit of an inconvenience, but nothing that can't be kill -9 'ed. Still should probably put a line in there to make sure it exits properly. Most distros won't stay on 4.0 forever, once they switch to newer versions the reports will come flooding in.
(0031812)
BillBlight (developer)
2017-04-30 08:33
edited on: 2017-04-30 08:48

After much searching in the googleverse, it seems a lot of mono developers are having issues with Environment.Exit, most are replacing it with alternative functions ..

Now for my purposes, I have replaced all instances of Environment.Exit, in ServicesServerBase.cs and BaseOpenSimServer.cs with System.Diagnostics.Process.GetCurrentProcess().Kill(); , now you tend to loose the ability to catch shutdown exceptions, but at this point I just want a clean shutdown and not a hang ...


And the instance still appears to finish all shutdown steps, it does not just die .. It dies at the end of shutting down the final thread .

(0031813)
tampa (reporter)
2017-04-30 13:50

Just tried that and shutdown works properly now. Probably not the actual way to do this in newer mono, but I haven't been able to find a proper fix for this either. Seems like a common issue with mono, but most just mention this to fix it. It shouldn't really matter much since it's pretty much the last step in the shutdown, but it's not issuing the same shutdown message as before so that might be an issue if that is used elsewhere.
(0031814)
BillBlight (developer)
2017-04-30 14:29

tampa, yeah I am sure someone may have a better "fix" but from what I have found there are lots of issues with "hangs" with Environment.Exit, in the mono world ..
(0031921)
BlueWall (administrator)
2017-05-22 18:12
edited on: 2017-05-23 03:40

Adding some information about this...

https://pastebin.com/QpsH3Y97 [^]

Mono JIT compiler version 4.6.2 (Stable 4.6.2.7/08fd525 Mon Nov 21 12:08:40 UTC 2016)

On this version of mono, one can enter the /proc/pid/task entry for the application and see separate pid for the hanging threads. Killing any of these allows the application to exit.

(0031928)
BlueWall (administrator)
2017-05-23 16:39

still hanging after update to 07e614a32c79b9295b162119d95ab1770243a8cc

Run in debugger - pause application execution manually after hangup and find the program looping here..

https://github.com/opensim/opensim/blob/master/OpenSim/Framework/BlockingQueue.cs#L75 [^]

Call stack...
System.Threading.Monitor.Monitor_wait() in
System.Threading.Monitor.ObjWait(bool exitContext, int millisecondsTimeout, object obj) in
System.Threading.Monitor.Wait(object obj, int millisecondsTimeout, bool exitContext) in
System.Threading.Monitor.Wait(object obj, int millisecondsTimeout) in
OpenSim.Framework.BlockingQueue<OpenSim.Framework.Servers.HttpServer.PollServiceHttpRequest>.Dequeue(int msTimeout) in /team/staging/opensim/OpenSim/Framework/BlockingQueue.cs:81
OpenSim.Framework.Servers.HttpServer.PollServiceRequestManager.PoolWorkerJob() in /team/staging/opensim/OpenSim/Framework/Servers/HttpServer/PollServiceRequestManager.cs:235
System.Threading.ThreadHelper.ThreadStart_Context(System.Threading.ThreadHelper state) in
System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Threading.ThreadHelper state, bool preserveSyncCtx) in
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Threading.ThreadHelper state, bool preserveSyncCtx) in
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Threading.ThreadHelper state) in
System.Threading.ThreadHelper.ThreadStart() in
(0031929)
UbitUmarov (administrator)
2017-05-23 17:15

that is a wait with a timeout of 5s
and now code does issue a abort to those threads
(0031930)
UbitUmarov (administrator)
2017-05-23 17:16

and by default it is background thread.. ie a non blocking one on exit
(0031931)
UbitUmarov (administrator)
2017-05-23 17:44

this is now the log output on windows shutdown:
the thread you seen above is one of the 3 PollServiceWorkerThread
note that most of this threads are background ones, we should not need to kill them (on hard quit at least)

---
2017-05-24 01:39:04,309 DEBUG [WATCHDOG]: Removing thread MapBlockSendThread (ubittest11), ID 41
2017-05-24 01:39:04,379 DEBUG [GetMeshModule] Closing
2017-05-24 01:39:04,399 DEBUG [WATCHDOG]: Removing thread GetMeshWorker0, ID 43
2017-05-24 01:39:04,414 DEBUG [WATCHDOG]: Removing thread GetMeshWorker1, ID 44
2017-05-24 01:39:04,414 DEBUG [GetTextureModule] Closing
2017-05-24 01:39:04,429 DEBUG [WATCHDOG]: Removing thread GetTextureWorker0, ID 45
2017-05-24 01:39:04,454 DEBUG [WATCHDOG]: Removing thread GetTextureWorker1, ID 46
2017-05-24 01:39:04,454 DEBUG [WebFetchInvDescModule] Closing
2017-05-24 01:39:04,469 DEBUG [WATCHDOG]: Removing thread InventoryWorkerThread0, ID 17
2017-05-24 01:39:04,484 DEBUG [WATCHDOG]: Removing thread InventoryWorkerThread1, ID 47
2017-05-24 01:39:04,484 INFO [LLUDPSERVER]: Shutting down the LLUDP server for ubittest13
2017-05-24 01:39:04,484 DEBUG [UDPBASE]: Stopping outbound UDP loop
2017-05-24 01:39:04,484 DEBUG [UDPBASE]: Stopping inbound UDP loop
2017-05-24 01:39:04,484 DEBUG [JobEngine] Stopping Incoming Packet Async Handling Engine (ubittest13)
2017-05-24 01:39:04,509 DEBUG [JobEngine] Stopping Outgoing Queue Refill Engine (ubittest13)
2017-05-24 01:39:04,529 DEBUG [WATCHDOG]: Removing thread MapItemRequestThread (ubittest13), ID 53
2017-05-24 01:39:04,534 DEBUG [JobEngine] Stopping HG Incoming Scene Object Engine (ubittest13)
2017-05-24 01:39:04,554 DEBUG [WATCHDOG]: Removing thread Outgoing Packets (ubittest13), ID 57
2017-05-24 01:39:04,579 INFO [XEngine]: Shutting down 96 scripts in ubittest13
2017-05-24 01:39:04,639 DEBUG [SCENE]: Dispose Physics
2017-05-24 01:39:04,724 DEBUG [WATCHDOG]: Removing thread Incoming Packets (ubittest13), ID 56
2017-05-24 01:39:04,744 DEBUG [JobEngine] Stopping ServiceThrottle
2017-05-24 01:39:05,174 DEBUG [WATCHDOG]: Removing thread PollServiceWorkerThread 0:9000, ID 21
2017-05-24 01:39:05,189 DEBUG [WATCHDOG]: Removing thread PollServiceWorkerThread 1:9000, ID 22
2017-05-24 01:39:06,824 DEBUG [WATCHDOG]: Removing thread PollServiceWatcherThread:9000, ID 23
2017-05-24 01:39:06,824 DEBUG [WATCHDOG]: Removing thread MapBlockSendThread (ubittest13), ID 54
2017-05-24 01:39:10,699 DEBUG [JobEngine] Stopping Non-blocking non-critical job engine
2017-05-24 01:39:10,719 INFO [SHUTDOWN]: Shutdown processing on main thread complete. Exiting...
(0031952)
BlueWall (administrator)
2017-05-24 09:03

8989e8ef Does complete the shutdown process on standalone. But, both Robust and OpenSim still hang on exit.
(0031958)
UbitUmarov (administrator)
2017-05-24 22:20

Did some changes, and now I do see shutdown work on more cases with mono 5.0
Possible still needs more work.

Thanks Bill Blight for setting up a Ubuntu VM just for me to test this issue
(0031959)
BlueWall (administrator)
2017-05-25 05:48

Latest:

OpenSim has nice clean shutdown,

Robust does shut down, but leaves this behind...

08:20:48 - [CONSOLE]: Quitting

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at OpenSim.Framework.Monitoring.JobEngine.Stop () [0x00099] in /team/staging/opensim/OpenSim/Framework/Monitoring/JobEngine.cs:144
  at OpenSim.Framework.Monitoring.WorkManager.Stop () [0x00001] in /team/staging/opensim/OpenSim/Framework/Monitoring/WorkManager.cs:87
  at OpenSim.Server.Base.ServicesServerBase.Run () [0x00050] in /team/staging/opensim/OpenSim/Server/Base/ServicesServerBase.cs:255
  at OpenSim.Server.OpenSimServer.Main (System.String[] args) [0x00330] in /team/staging/opensim/OpenSim/Server/ServerMain.cs:163
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at OpenSim.Framework.Monitoring.JobEngine.Stop () [0x00099] in /team/staging/opensim/OpenSim/Framework/Monitoring/JobEngine.cs:144
  at OpenSim.Framework.Monitoring.WorkManager.Stop () [0x00001] in /team/staging/opensim/OpenSim/Framework/Monitoring/WorkManager.cs:87
  at OpenSim.Server.Base.ServicesServerBase.Run () [0x00050] in /team/staging/opensim/OpenSim/Server/Base/ServicesServerBase.cs:255
  at OpenSim.Server.OpenSimServer.Main (System.String[] args) [0x00330] in /team/staging/opensim/OpenSim/Server/ServerMain.cs:163
(0031960)
BillBlight (developer)
2017-05-25 07:20

Shuts down fine on regions as above, but now seeing this error

10:16:00 - Timeout in threadfunc 1 (STP:Util:49)
10:16:00 - Timeout in threadfunc 2 (STP:Util:21)
10:16:00 - Timeout in threadfunc 3 (STP:Util:24)
10:16:00 - Timeout in threadfunc 4 (STP:Util:45)

The last number is usually different .. Not sure if it is related to the shutdown or not, this error is actually on startup, and occasionally. But just in this latest build.

(And you are very welcome for the server, have fun with it)
(0031961)
UbitUmarov (administrator)
2017-05-25 17:08

I did fix those timeouts.. on my box.. missed the file on commit to master :)
(0031968)
Marcus Llewellyn (reporter)
2017-05-26 09:56

I can confirm clean shutdowns now.

Mono JIT compiler version 5.0.0 (Stable 5.0.0.100/9667aa6 Tue May 23 12:12:24 CDT 2017)

Compiled from commit 8f10db0a6a4831cded816e6d08b09fe0e47c77b4

Ubuntu 17.04 x86_64 Linux 4.10.0-21-generic
(0031969)
UbitUmarov (administrator)
2017-05-26 15:02

Compile as Release does help a bit on all aspects

Just if something goes wrong, there is less information to debug it
(0032105)
slow putzo (reporter)
2017-07-01 13:16
edited on: 2017-07-01 13:18

Confirm Fedora 25 with:
Mono JIT compiler version 4.4.2 (Stable 4.4.2.11/f72fe45 Tue Aug 2 08:14:37 UTC 2016)

and:
Mono JIT compiler version 5.0.1.1 (2017-02/5077205 Wed May 24 13:16:08 UTC 2017)

Clean shutdown.

(0032798)
Ravelli Ormstein (reporter)
2018-07-13 19:18

Having the same issue now, but under Windows 7 64 bit and the official OpenSimulator release 0.9.0.1. Everything works fine, when I launch OpenSimulator and shut it down soon again. But after having it running for several hours, it does not shut down anymore.

For shutting down OpenSimulator I use the 'q' command. Closing the process does not work, not even via 'Kill Process'. I have to restart the entire system to get rid of the process. It is a stand-alone grid with HG enabled.

The update log of Windows lists an update on 11th July for NET Framework 3.5.1, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, 4.7.1, 4.7.2 under Windows 7 and Server 2008 R2 for x64 (KB4340556) - http://support.microsoft.com/kb/4340556 [^]

Site note: I have the same issue with my Apache server, which I use occasionally, not 24/7 - (httpd.exe on port 80)

- Issue History
Date Modified Username Field Change
2015-11-18 08:09 Shy Robbiani New Issue
2015-11-18 18:19 OtakuMegane Note Added: 0029563
2015-11-19 05:51 mewtwo0641 Note Added: 0029567
2015-11-19 05:51 mewtwo0641 Note Edited: 0029567 View Revisions
2015-11-19 05:59 Jak Daniels Note Added: 0029568
2015-11-19 07:49 UbitUmarov Note Added: 0029570
2015-11-19 08:16 zadark Note Added: 0029572
2015-11-19 08:17 zadark Note Edited: 0029572 View Revisions
2015-11-19 08:22 zadark Note Edited: 0029572 View Revisions
2015-11-19 08:23 zadark Note Edited: 0029572 View Revisions
2015-11-19 08:42 Jak Daniels Note Added: 0029573
2015-11-19 08:56 Jak Daniels Note Edited: 0029573 View Revisions
2015-11-19 09:13 Jak Daniels Note Edited: 0029573 View Revisions
2015-11-19 16:58 UbitUmarov Note Added: 0029575
2015-11-21 05:35 Shy Robbiani Note Added: 0029593
2015-11-23 20:07 mewtwo0641 Note Added: 0029621
2015-11-24 09:28 Shy Robbiani Note Added: 0029638
2015-11-24 09:28 Shy Robbiani Status new => resolved
2015-11-24 09:28 Shy Robbiani Resolution open => no change required
2015-11-24 09:28 Shy Robbiani Assigned To => Shy Robbiani
2015-11-26 07:58 slow putzo Note Added: 0029684
2015-11-26 07:58 slow putzo Status resolved => acknowledged
2015-11-26 13:05 slow putzo Note Added: 0029699
2015-11-27 01:57 aiaustin Note Added: 0029717
2015-11-27 01:59 aiaustin Note Edited: 0029717 View Revisions
2015-11-27 02:18 aiaustin Note Edited: 0029717 View Revisions
2015-11-27 02:29 aiaustin Note Edited: 0029717 View Revisions
2015-11-27 02:34 aiaustin Note Edited: 0029717 View Revisions
2015-11-27 07:33 UbitUmarov Note Added: 0029720
2015-11-27 08:01 aiaustin Note Added: 0029721
2015-11-27 12:50 slow putzo Note Added: 0029725
2015-11-27 14:09 aiaustin Note Added: 0029726
2015-11-27 14:11 aiaustin Note Edited: 0029726 View Revisions
2015-11-27 14:12 aiaustin Note Edited: 0029726 View Revisions
2015-11-27 14:12 aiaustin Note Edited: 0029726 View Revisions
2015-11-27 14:14 aiaustin Note Edited: 0029726 View Revisions
2015-11-27 14:18 aiaustin Note Edited: 0029726 View Revisions
2015-11-27 14:22 aiaustin Note Edited: 0029726 View Revisions
2015-11-27 15:03 slow putzo Note Added: 0029728
2015-11-27 15:21 UbitUmarov Note Added: 0029729
2015-11-27 15:28 slow putzo Note Added: 0029730
2015-11-27 16:23 slow putzo Note Added: 0029732
2015-11-27 16:28 UbitUmarov Note Added: 0029733
2015-11-27 16:39 OtakuMegane Note Added: 0029734
2015-11-27 16:47 slow putzo Note Added: 0029735
2015-11-27 16:49 UbitUmarov Note Added: 0029736
2015-11-27 16:59 OtakuMegane Note Added: 0029737
2015-11-27 17:00 UbitUmarov Note Added: 0029738
2015-11-27 17:01 slow putzo Note Added: 0029739
2015-11-28 01:21 aiaustin Note Added: 0029744
2015-11-28 01:22 aiaustin Note Edited: 0029744 View Revisions
2015-11-28 01:24 aiaustin Note Edited: 0029744 View Revisions
2015-11-28 06:37 aiaustin Note Edited: 0029726 View Revisions
2015-11-28 09:35 aiaustin Note Edited: 0029726 View Revisions
2015-11-28 09:36 aiaustin Note Edited: 0029726 View Revisions
2015-12-11 16:49 slow putzo Note Added: 0029802
2015-12-19 15:19 OtakuMegane Note Added: 0029835
2015-12-21 19:54 slow putzo Note Added: 0029847
2015-12-22 03:20 UbitUmarov Note Added: 0029849
2015-12-27 21:31 Robert Adams Note Added: 0029893
2017-04-08 09:49 slow putzo Note Added: 0031731
2017-04-08 09:49 slow putzo File Added: OpenSim.log
2017-04-08 10:06 slow putzo Note Added: 0031732
2017-04-08 10:17 nebadon Note Added: 0031733
2017-04-08 10:19 slow putzo File Deleted: OpenSim.log
2017-04-08 10:20 slow putzo File Added: OpenSim.log
2017-04-08 10:29 slow putzo Note Added: 0031734
2017-04-08 10:31 nebadon Note Added: 0031735
2017-04-08 10:32 nebadon Note Added: 0031736
2017-04-08 10:43 slow putzo Note Added: 0031737
2017-04-08 10:44 nebadon Note Added: 0031738
2017-04-08 10:52 slow putzo Note Added: 0031739
2017-04-08 10:56 aiaustin Note Added: 0031740
2017-04-08 10:57 aiaustin Note Edited: 0031740 View Revisions
2017-04-09 13:34 tampa Note Added: 0031745
2017-04-30 08:33 BillBlight Note Added: 0031812
2017-04-30 08:33 BillBlight Note Edited: 0031812 View Revisions
2017-04-30 08:48 BillBlight Note Edited: 0031812 View Revisions
2017-04-30 13:50 tampa Note Added: 0031813
2017-04-30 14:29 BillBlight Note Added: 0031814
2017-05-22 18:12 BlueWall Note Added: 0031921
2017-05-22 18:13 BlueWall Note Edited: 0031921 View Revisions
2017-05-22 18:19 BlueWall Note Edited: 0031921 View Revisions
2017-05-22 18:22 BlueWall Note Edited: 0031921 View Revisions
2017-05-23 03:40 BlueWall Note Edited: 0031921 View Revisions
2017-05-23 16:39 BlueWall Note Added: 0031928
2017-05-23 17:15 UbitUmarov Note Added: 0031929
2017-05-23 17:16 UbitUmarov Note Added: 0031930
2017-05-23 17:44 UbitUmarov Note Added: 0031931
2017-05-24 01:00 aiaustin Note Deleted: 0031740
2017-05-24 09:03 BlueWall Note Added: 0031952
2017-05-24 22:20 UbitUmarov Note Added: 0031958
2017-05-25 05:48 BlueWall Note Added: 0031959
2017-05-25 07:20 BillBlight Note Added: 0031960
2017-05-25 17:08 UbitUmarov Note Added: 0031961
2017-05-26 09:56 Marcus Llewellyn Note Added: 0031968
2017-05-26 15:02 UbitUmarov Note Added: 0031969
2017-07-01 13:16 slow putzo Note Added: 0032105
2017-07-01 13:18 slow putzo Note Edited: 0032105 View Revisions
2018-07-13 19:18 Ravelli Ormstein Note Added: 0032798


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker