MantisBT - opensim
View Issue Details
0008183opensim[REGION] OpenSim Corepublic2017-06-02 01:592017-06-10 05:41
aiaustin 
aiaustin 
normalminoralways
closedfixed 
PCWindows10
master (dev code) 
master (dev code)master (dev code) 
opensim-0.9.1-1426-g8f10db0
Grid (Multiple Regions per Sim)
BulletSim
.NET / Windows64
None
N/A
0008183: [WATCHDOG]: Removing thread BackupWaitCallback - but this is not being monitored
In a recent update of the dev master code on my grids (Openvue and AiLand) I have started to see a warning from [WACTCHDOG] with pairs of messages of this form... one per region on the region OpenSim.exe console.

13:15:04 - [WATCHDOG]: Removing thread BackupWaitCallback (Castle), ID 152
13:15:04 - [WATCHDOG]: Requested to remove thread with ID 152 but this is not being monitored

13:31:02 - [WATCHDOG]: Removing thread BackupWaitCallback (Aisle), ID 160
13:31:02 - [WATCHDOG]: Requested to remove thread with ID 160 but this is not being monitored

13:45:59 - [WATCHDOG]: Removing thread BackupWaitCallback (Black Rock), ID 72
13:45:59 - [WATCHDOG]: Requested to remove thread with ID 72 but this is not being monitored
No tags attached.
Issue History
2017-06-02 01:59aiaustinNew Issue
2017-06-02 02:03aiaustinNote Added: 0031990
2017-06-02 09:43aiaustinNote Edited: 0031990bug_revision_view_page.php?bugnote_id=31990#r6226
2017-06-07 02:13aiaustinNote Added: 0031997
2017-06-07 02:14aiaustinNote Edited: 0031997bug_revision_view_page.php?bugnote_id=31997#r6228
2017-06-07 02:14aiaustinNote Edited: 0031997bug_revision_view_page.php?bugnote_id=31997#r6229
2017-06-09 02:49aiaustinNote Added: 0032015
2017-06-09 02:50aiaustinNote Edited: 0032015bug_revision_view_page.php?bugnote_id=32015#r6244
2017-06-09 03:42UbitUmarovNote Added: 0032017
2017-06-09 08:24aiaustinNote Edited: 0032015bug_revision_view_page.php?bugnote_id=32015#r6248
2017-06-09 10:18BillBlightNote Added: 0032018
2017-06-09 10:52aiaustinNote Added: 0032019
2017-06-09 11:03UbitUmarovNote Added: 0032020
2017-06-09 11:06BillBlightNote Added: 0032021
2017-06-09 11:33aiaustinNote Added: 0032022
2017-06-10 05:41aiaustinNote Added: 0032024
2017-06-10 05:41aiaustinStatusnew => resolved
2017-06-10 05:41aiaustinFixed in Version => master (dev code)
2017-06-10 05:41aiaustinResolutionopen => fixed
2017-06-10 05:41aiaustinAssigned To => aiaustin
2017-06-10 05:41aiaustinStatusresolved => closed

Notes
(0031990)
aiaustin   
2017-06-02 02:03   
(edited on: 2017-06-02 09:43)
I notice that its also on CleanTempObjects...

08:46:56 - [WATCHDOG]: Removing thread CleanTempObjects (OpenVCE), ID 92
08:46:56 - [WATCHDOG]: Requested to remove thread with ID 92 but this is not being monitored
12:41:34 - [WATCHDOG]: Removing thread CleanTempObjects (OpenVCE), ID 23
12:41:34 - [WATCHDOG]: Requested to remove thread with ID 23 but this is not being monitored

(0031997)
aiaustin   
2017-06-07 02:13   
(edited on: 2017-06-07 02:14)
I checked the times at which the issues were arising for any one region and cannot see a pattern. Some are 30 minutes apart, some over 4 hours apart. On OpenSim.exe region server consoles the messages are filling the screen...

I think that the issue was introduced somewhere between

opensim-0.9.0-1408-g07e614a.zip (2017-05-25)

opensim-0.9.0-1426-g8f10db0.zip (2017-05-26)

There were some changes to thread handling code around that time.

(0032015)
aiaustin   
2017-06-09 02:49   
(edited on: 2017-06-09 08:24)
I saw some recent changes to thread handling again, but the latest dev master does still show a stream of OpenSim.exe console messages about this for BackupWorker and CleanTempObjects... on what is essentially an idling region server at times. E.g.

19:58:24 - [WATCHDOG]: Removing thread BackupWorker (Fallingwater), ID 271
19:58:24 - [WATCHDOG]: Requested to remove thread with ID 271 but this is not being monitored
20:05:38 - [WATCHDOG]: Removing thread CleanTempObjects (Marineville), ID 162
20:05:38 - [WATCHDOG]: Requested to remove thread with ID 162 but this is not being monitored
20:32:53 - [WATCHDOG]: Removing thread CleanTempObjects (Fallingwater), ID 265
20:32:53 - [WATCHDOG]: Requested to remove thread with ID 265 but this is not being monitored
21:32:56 - [WATCHDOG]: Removing thread BackupWorker (Fallingwater), ID 244
21:32:56 - [WATCHDOG]: Requested to remove thread with ID 244 but this is not being monitored
21:39:31 - [WATCHDOG]: Removing thread BackupWorker (Space City), ID 254
21:39:31 - [WATCHDOG]: Requested to remove thread with ID 254 but this is not being monitored
21:59:00 - [WATCHDOG]: Removing thread BackupWorker (Fallingwater), ID 221
21:59:00 - [WATCHDOG]: Requested to remove thread with ID 221 but this is not being monitored
22:15:05 - [WATCHDOG]: Removing thread CleanTempObjects (Aisle), ID 190
22:15:05 - [WATCHDOG]: Requested to remove thread with ID 190 but this is not being monitored
00:23:06 - [WATCHDOG]: Removing thread BackupWorker (Aisle), ID 228
00:23:06 - [WATCHDOG]: Requested to remove thread with ID 228 but this is not being monitored
00:29:06 - [WATCHDOG]: Removing thread CleanTempObjects (AiAi), ID 196
00:29:06 - [WATCHDOG]: Requested to remove thread with ID 196 but this is not being monitored
04:44:51 - [WATCHDOG]: Removing thread BackupWorker (Black Rock), ID 255
04:44:51 - [WATCHDOG]: Requested to remove thread with ID 255 but this is not being monitored
04:50:45 - [WATCHDOG]: Removing thread BackupWorker (Marineville), ID 176
04:50:45 - [WATCHDOG]: Requested to remove thread with ID 176 but this is not being monitored
07:19:51 - [WATCHDOG]: Removing thread BackupWorker (Castle), ID 179
07:19:51 - [WATCHDOG]: Requested to remove thread with ID 179 but this is not being monitored
08:03:11 - [WATCHDOG]: Removing thread CleanTempObjects (Castle), ID 232
08:03:11 - [WATCHDOG]: Requested to remove thread with ID 232 but this is not being monitored
10:28:33 - [WATCHDOG]: Removing thread BackupWorker (Space City), ID 165
10:28:33 - [WATCHDOG]: Requested to remove thread with ID 165 but this is not being monitored

(0032017)
UbitUmarov   
2017-06-09 03:42   
annoying and my regions don't want to show it :)
(0032018)
BillBlight   
2017-06-09 10:18   
I don't see it either on instances with a single region, but on a instance with more than a couple of regions I do see it once in a while ..
(0032019)
aiaustin   
2017-06-09 10:52   
My region severs have three, six, eight and eight regions in them. All on Xeon 8 core, 32GB memory servers. The messages only started when I reported. Not seen prior to that in several years constant running with similar configurations.
(0032020)
UbitUmarov   
2017-06-09 11:03   
yes think it started showing up when i changed the detection of stopped threads
and silent removal of them from watchdog list
SHA-1: d90b68c2a637b822b826a0cf3c52991aa9ee2c97

* change wrong watchdog stopped thread check code, and don't log it

if(threadInfo.Thread.ThreadState == ThreadState.Stopped) was wrong bc ThreadState is a bitfield that does include background for example, so was never true on this threads.

well on master those are now out of that code, not because of this but because those where new threads requested from system, and they should came from the pool, since, well that's the purpose of having a thread pool..
(0032021)
BillBlight   
2017-06-09 11:06   
Yeah I went back through my logs it is there , just harder to catch with only one region .. I should have done a search of the logs before I posted .. But not getting it now ..
(0032022)
aiaustin   
2017-06-09 11:33   
Great... I updated Openvue and AiLand grids to OpenSim 0.9.1 1478 f69e48b (2017-06-09 16:32) and will watch the logs over the next 24 hours.
(0032024)
aiaustin   
2017-06-10 05:41   
No warnings are reported on two test grids with multiple addon regions on several servers on each grid.