Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007751opensim[REGION] OpenSim Corepublic2015-11-23 07:412015-11-24 02:44
Reporteraiaustin 
Assigned To 
PrioritynormalSeverityminorReproducibilityalways
StatusnewResolutionopen 
PlatformPCOSWindowsOS Version10
Product Versionmaster (dev code) 
Target Versionmaster (dev code)Fixed in Version 
Summary0007751: Four new watchdog timeouts for each region on server shutdown
DescriptionUsing the latest dev master I am seeing a lot of WATCHDOG timeout issues.... and I get a set of these on shutdown of the OpenSim.exe region server... four for each of the regions in that instance. Each reports that the ThreadState=Stopped when the timeouts occur.

As an example, on region "Vue-Point" there are timeouts on these 4 threads...

HG Incoming Scene Object Engine (Vue-Point)
Incoming Packet Async Handling Engine (Vue-Point)
Outgoing Queue Refill Engine (Vue-Point)
Heartbeat-(Vue-Port)

There is no such issue on the version I was using prior to the avination code merge... e.g. opensim-2015-11-13-02-40-554d4ba (opensim-554d4ba.zip)

[WATCHDOG]: Timeout detected for thread "HG Incoming Scene Object Engine (Vue-Point)". ThreadState=Stopped. Last tick was 403562ms ago.

[WATCHDOG]: Timeout detected for thread "Incoming Packet Async Handling Engine (Vue-Point)". ThreadState=Stopped. Last tick was 403546ms ago.

[WATCHDOG]: Timeout detected for thread "Outgoing Queue Refill Engine (Vue-Point)". ThreadState=Stopped. Last tick was 403546ms ago.

[WATCHDOG]: Timeout detected for thread "Heartbeat-(Vue-Point)". ThreadState=Stopped. Last tick was 218ms ago.
Steps To ReproduceUsing opensim-0.9.0-86-g15af626.zip (2015-11-22 16:02) on Windows 10.

This occurs without any client/user login.. It shows even if you just start Robvust.exe and OpenSim.exe on one server. Then shutdown.

Reverting to a version from 10 days ago and these issues do not show. E,g, on opensim-2015-11-13-02-40-554d4ba (opensim-554d4ba.zip)
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Environment.NET / Windows64
Mono VersionNone
ViewerN/A
Attached Files

- Relationships

-  Notes
(0029606)
aiaustin (developer)
2015-11-23 08:32
edited on: 2015-11-23 10:09

Using opensim-0.9.0-86-g15af626.zip (2015-11-22 16:02) on Windows 10 on a system with nothing else running...

Even on a simple Standalone OpenSim.exe with one pimple island region, one (Ruth) avatar and no prims... I started the server and then after a minute or two shut it down with no logins in between and it gave a watchdog timeout error.

ERROR (18) - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat-(Vue)". ThreadState=Stopped. Last tick was 328ms ago

Revertng to version before avination merge and this goes away.

(0029607)
Garmin Kawaguichi (reporter)
2015-11-23 08:38

To complete aiaustin mantis:

Found in Opensim.log:
---------------------------
2015-11-18 00:23:25,054 DEBUG - OpenSim.Framework.Monitoring.Watchdog [WATCHDOG]: Removing thread MapBlockSendThread (Vide3), ID 48
{…}
2015-11-18 00:23:25,054 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "MapBlockSendThread (Vide3)". ThreadState=Stopped. Last tick was 1115047ms ago.
---------------------------
1115047ms = 18 minutes 35 seconds
---------------------------
ThreadState=Stopped
This sounds like the removing fails or is wrong or is incomplete.
---------------------------
When there are around fifteen regions that stop at the same time, this is a bloody tsunami wave on the console.

MML
(0029624)
aiaustin (developer)
2015-11-24 02:44
edited on: 2015-11-24 02:44

Some of the timeout "last tick" figures are incredible... here is one on shutdown after leaving the server running overnight. Incoming Scene Object Engine (Vue-Point)". ThreadState=Stopped. Last tick was 68439204ms = 19 hours.


[WATCHDOG]: Timeout detected for thread "HG Incoming Scene Object Engine (Vue-Point)". ThreadState=Stopped. Last tick was 68439204ms ago.

[WATCHDOG]: Timeout detected for thread "Incoming Packet Async Handling Engine (Vue-Point)". ThreadState=Stopped. Last tick was 68439188ms ago.

[WATCHDOG]: Timeout detected for thread "Outgoing Queue Refill Engine (Vue-Point)". ThreadState=Stopped. Last tick was 68439188ms ago.

[WATCHDOG]: Timeout detected for thread "Heartbeat-(Vue-Point)". ThreadState=Stopped. Last tick was 2313ms ago.


- Issue History
Date Modified Username Field Change
2015-11-23 07:41 aiaustin New Issue
2015-11-23 07:41 aiaustin Description Updated View Revisions
2015-11-23 08:32 aiaustin Note Added: 0029606
2015-11-23 08:33 aiaustin Note Edited: 0029606 View Revisions
2015-11-23 08:38 Garmin Kawaguichi Note Added: 0029607
2015-11-23 10:05 aiaustin Description Updated View Revisions
2015-11-23 10:08 aiaustin Description Updated View Revisions
2015-11-23 10:08 aiaustin Description Updated View Revisions
2015-11-23 10:09 aiaustin Note Edited: 0029606 View Revisions
2015-11-24 02:44 aiaustin Note Added: 0029624
2015-11-24 02:44 aiaustin Note Edited: 0029624 View Revisions


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker