Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008183opensim[REGION] OpenSim Corepublic2017-06-02 01:592017-06-10 05:41
Reporteraiaustin 
Assigned Toaiaustin 
PrioritynormalSeverityminorReproducibilityalways
StatusclosedResolutionfixed 
PlatformPCOSWindowsOS Version10
Product Versionmaster (dev code) 
Target Versionmaster (dev code)Fixed in Versionmaster (dev code) 
Summary0008183: [WATCHDOG]: Removing thread BackupWaitCallback - but this is not being monitored
DescriptionIn a recent update of the dev master code on my grids (Openvue and AiLand) I have started to see a warning from [WACTCHDOG] with pairs of messages of this form... one per region on the region OpenSim.exe console.

13:15:04 - [WATCHDOG]: Removing thread BackupWaitCallback (Castle), ID 152
13:15:04 - [WATCHDOG]: Requested to remove thread with ID 152 but this is not being monitored

13:31:02 - [WATCHDOG]: Removing thread BackupWaitCallback (Aisle), ID 160
13:31:02 - [WATCHDOG]: Requested to remove thread with ID 160 but this is not being monitored

13:45:59 - [WATCHDOG]: Removing thread BackupWaitCallback (Black Rock), ID 72
13:45:59 - [WATCHDOG]: Requested to remove thread with ID 72 but this is not being monitored
TagsNo tags attached.
Git Revision or version numberopensim-0.9.1-1426-g8f10db0
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Environment.NET / Windows64
Mono VersionNone
ViewerN/A
Attached Files

- Relationships

-  Notes
(0031990)
aiaustin (developer)
2017-06-02 02:03
edited on: 2017-06-02 09:43

I notice that its also on CleanTempObjects...

08:46:56 - [WATCHDOG]: Removing thread CleanTempObjects (OpenVCE), ID 92
08:46:56 - [WATCHDOG]: Requested to remove thread with ID 92 but this is not being monitored
12:41:34 - [WATCHDOG]: Removing thread CleanTempObjects (OpenVCE), ID 23
12:41:34 - [WATCHDOG]: Requested to remove thread with ID 23 but this is not being monitored

(0031997)
aiaustin (developer)
2017-06-07 02:13
edited on: 2017-06-07 02:14

I checked the times at which the issues were arising for any one region and cannot see a pattern. Some are 30 minutes apart, some over 4 hours apart. On OpenSim.exe region server consoles the messages are filling the screen...

I think that the issue was introduced somewhere between

opensim-0.9.0-1408-g07e614a.zip (2017-05-25)

opensim-0.9.0-1426-g8f10db0.zip (2017-05-26)

There were some changes to thread handling code around that time.

(0032015)
aiaustin (developer)
2017-06-09 02:49
edited on: 2017-06-09 08:24

I saw some recent changes to thread handling again, but the latest dev master does still show a stream of OpenSim.exe console messages about this for BackupWorker and CleanTempObjects... on what is essentially an idling region server at times. E.g.

19:58:24 - [WATCHDOG]: Removing thread BackupWorker (Fallingwater), ID 271
19:58:24 - [WATCHDOG]: Requested to remove thread with ID 271 but this is not being monitored
20:05:38 - [WATCHDOG]: Removing thread CleanTempObjects (Marineville), ID 162
20:05:38 - [WATCHDOG]: Requested to remove thread with ID 162 but this is not being monitored
20:32:53 - [WATCHDOG]: Removing thread CleanTempObjects (Fallingwater), ID 265
20:32:53 - [WATCHDOG]: Requested to remove thread with ID 265 but this is not being monitored
21:32:56 - [WATCHDOG]: Removing thread BackupWorker (Fallingwater), ID 244
21:32:56 - [WATCHDOG]: Requested to remove thread with ID 244 but this is not being monitored
21:39:31 - [WATCHDOG]: Removing thread BackupWorker (Space City), ID 254
21:39:31 - [WATCHDOG]: Requested to remove thread with ID 254 but this is not being monitored
21:59:00 - [WATCHDOG]: Removing thread BackupWorker (Fallingwater), ID 221
21:59:00 - [WATCHDOG]: Requested to remove thread with ID 221 but this is not being monitored
22:15:05 - [WATCHDOG]: Removing thread CleanTempObjects (Aisle), ID 190
22:15:05 - [WATCHDOG]: Requested to remove thread with ID 190 but this is not being monitored
00:23:06 - [WATCHDOG]: Removing thread BackupWorker (Aisle), ID 228
00:23:06 - [WATCHDOG]: Requested to remove thread with ID 228 but this is not being monitored
00:29:06 - [WATCHDOG]: Removing thread CleanTempObjects (AiAi), ID 196
00:29:06 - [WATCHDOG]: Requested to remove thread with ID 196 but this is not being monitored
04:44:51 - [WATCHDOG]: Removing thread BackupWorker (Black Rock), ID 255
04:44:51 - [WATCHDOG]: Requested to remove thread with ID 255 but this is not being monitored
04:50:45 - [WATCHDOG]: Removing thread BackupWorker (Marineville), ID 176
04:50:45 - [WATCHDOG]: Requested to remove thread with ID 176 but this is not being monitored
07:19:51 - [WATCHDOG]: Removing thread BackupWorker (Castle), ID 179
07:19:51 - [WATCHDOG]: Requested to remove thread with ID 179 but this is not being monitored
08:03:11 - [WATCHDOG]: Removing thread CleanTempObjects (Castle), ID 232
08:03:11 - [WATCHDOG]: Requested to remove thread with ID 232 but this is not being monitored
10:28:33 - [WATCHDOG]: Removing thread BackupWorker (Space City), ID 165
10:28:33 - [WATCHDOG]: Requested to remove thread with ID 165 but this is not being monitored

(0032017)
UbitUmarov (administrator)
2017-06-09 03:42

annoying and my regions don't want to show it :)
(0032018)
watcher64 (reporter)
2017-06-09 10:18

I don't see it either on instances with a single region, but on a instance with more than a couple of regions I do see it once in a while ..
(0032019)
aiaustin (developer)
2017-06-09 10:52

My region severs have three, six, eight and eight regions in them. All on Xeon 8 core, 32GB memory servers. The messages only started when I reported. Not seen prior to that in several years constant running with similar configurations.
(0032020)
UbitUmarov (administrator)
2017-06-09 11:03

yes think it started showing up when i changed the detection of stopped threads
and silent removal of them from watchdog list
SHA-1: d90b68c2a637b822b826a0cf3c52991aa9ee2c97

* change wrong watchdog stopped thread check code, and don't log it

if(threadInfo.Thread.ThreadState == ThreadState.Stopped) was wrong bc ThreadState is a bitfield that does include background for example, so was never true on this threads.

well on master those are now out of that code, not because of this but because those where new threads requested from system, and they should came from the pool, since, well that's the purpose of having a thread pool..
(0032021)
watcher64 (reporter)
2017-06-09 11:06

Yeah I went back through my logs it is there , just harder to catch with only one region .. I should have done a search of the logs before I posted .. But not getting it now ..
(0032022)
aiaustin (developer)
2017-06-09 11:33

Great... I updated Openvue and AiLand grids to OpenSim 0.9.1 1478 f69e48b (2017-06-09 16:32) and will watch the logs over the next 24 hours.
(0032024)
aiaustin (developer)
2017-06-10 05:41

No warnings are reported on two test grids with multiple addon regions on several servers on each grid.

- Issue History
Date Modified Username Field Change
2017-06-02 01:59 aiaustin New Issue
2017-06-02 02:03 aiaustin Note Added: 0031990
2017-06-02 09:43 aiaustin Note Edited: 0031990 View Revisions
2017-06-07 02:13 aiaustin Note Added: 0031997
2017-06-07 02:14 aiaustin Note Edited: 0031997 View Revisions
2017-06-07 02:14 aiaustin Note Edited: 0031997 View Revisions
2017-06-09 02:49 aiaustin Note Added: 0032015
2017-06-09 02:50 aiaustin Note Edited: 0032015 View Revisions
2017-06-09 03:42 UbitUmarov Note Added: 0032017
2017-06-09 08:24 aiaustin Note Edited: 0032015 View Revisions
2017-06-09 10:18 watcher64 Note Added: 0032018
2017-06-09 10:52 aiaustin Note Added: 0032019
2017-06-09 11:03 UbitUmarov Note Added: 0032020
2017-06-09 11:06 watcher64 Note Added: 0032021
2017-06-09 11:33 aiaustin Note Added: 0032022
2017-06-10 05:41 aiaustin Note Added: 0032024
2017-06-10 05:41 aiaustin Status new => resolved
2017-06-10 05:41 aiaustin Fixed in Version => master (dev code)
2017-06-10 05:41 aiaustin Resolution open => fixed
2017-06-10 05:41 aiaustin Assigned To => aiaustin
2017-06-10 05:41 aiaustin Status resolved => closed


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker