Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006178opensim[REGION] OpenSim Corepublic2012-05-07 02:242013-05-13 16:01
Reporterkenvc 
Assigned Tojustincc 
PrioritynormalSeverityminorReproducibilitysometimes
StatusclosedResolutionfixed 
PlatformQuad core 8 gig ramOSWindows 7OS Version64 Bit pro
Product Versionmaster (dev code) 
Target VersionFixed in Versionmaster (dev code) 
Summary0006178: [WATCHDOG] Timeout detected for thread ... Last tick was 0ms ago
DescriptionSeeing many red Errors regarding Watchdog timeouts on many different kinds of threads. Weird thing is some are showing a timeout after 0ms. How can something have a 0ms timeout???

This happens after startup and when nothing is going on in the sim. It has been happening off and on in different sims but definatly seems to be worse lately with multiple timeouts often happening together and shown below.

See section of log file included in Additional Information box below, and also attachment for log file from an entire day on a multiple sim instance that was not busy.
Steps To ReproduceRun opensim and sit and wait
Additional Information2012-05-07 03:07:35,546 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "MapItemRequestThread (Fantasy Dream 5)". ThreadState=Background, WaitSleepJoin. Last tick was 6521ms ago
2012-05-07 03:07:35,656 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Incoming Packets (Fantasy Dream 5)". ThreadState=WaitSleepJoin. Last tick was 0ms ago
2012-05-07 03:07:35,656 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Outgoing Packets (Fantasy Dream 5)". ThreadState=WaitSleepJoin. Last tick was 0ms ago
2012-05-07 03:07:35,666 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat (Fantasy Dream 5)". ThreadState=WaitSleepJoin. Last tick was 47ms ago
2012-05-07 03:07:35,666 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "MapItemRequestThread (Fantasy Dream 6)". ThreadState=Background, WaitSleepJoin. Last tick was 218ms ago
2012-05-07 03:07:35,666 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Incoming Packets (Fantasy Dream 6)". ThreadState=WaitSleepJoin. Last tick was 16ms ago
2012-05-07 03:07:35,666 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Outgoing Packets (Fantasy Dream 6)". ThreadState=WaitSleepJoin. Last tick was 16ms ago
2012-05-07 03:07:35,676 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "MapItemRequestThread (Fantasy Dream 7)". ThreadState=Background, WaitSleepJoin. Last tick was 234ms ago
2012-05-07 03:07:35,676 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "AsyncLSLCmdHandlerThread". ThreadState=Background, WaitSleepJoin. Last tick was 32ms ago
2012-05-07 03:07:35,676 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Incoming Packets (Fantasy Dream 7)". ThreadState=WaitSleepJoin. Last tick was 32ms ago
2012-05-07 03:07:35,686 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Outgoing Packets (Fantasy Dream 7)". ThreadState=WaitSleepJoin. Last tick was 0ms ago
2012-05-07 03:07:35,686 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat (Fantasy Dream 7)". ThreadState=WaitSleepJoin. Last tick was 63ms ago
2012-05-07 03:07:35,686 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "MapItemRequestThread (Fantasy Dream 8)". ThreadState=Background, WaitSleepJoin. Last tick was 234ms ago
2012-05-07 03:07:35,686 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Maintenance (Fantasy Dream 6)". ThreadState=WaitSleepJoin. Last tick was 234ms ago
2012-05-07 03:07:35,696 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Incoming Packets (Fantasy Dream 8)". ThreadState=WaitSleepJoin. Last tick was 47ms ago
2012-05-07 03:07:35,696 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Outgoing Packets (Fantasy Dream 8)". ThreadState=WaitSleepJoin. Last tick was 0ms ago
2012-05-07 03:07:35,696 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Heartbeat (Fantasy Dream 8)". ThreadState=WaitSleepJoin. Last tick was 78ms ago
2012-05-07 03:07:35,706 ERROR - OpenSim.OpenSim [WATCHDOG]: Timeout detected for thread "Maintenance (Fantasy Dream 8)". ThreadState=WaitSleepJoin. Last tick was 265ms ago
TagsNo tags attached.
Git Revision or version number18912
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBasicPhysics
Environment.NET / Windows64
Mono VersionNone
ViewerN/A
Attached Files? file icon OpenSim.32BitLaunch.log.2012-05-10 [^] (178,409 bytes) 2012-05-12 06:27

- Relationships

-  Notes
(0022119)
justincc (administrator)
2012-05-11 19:16

How long after sim startup do these messages occur? Can you attach the portions of the log showing the timestamps?
(0022120)
kenvc (reporter)
2012-05-11 21:06

Justin,

These show up right after I see the message about logins being enabled, They may happen other times too but that's always when I notice them. I'll try to do a clean start-up tomorrow and send a larger hunk of the log file.
(0022121)
kenvc (reporter)
2012-05-12 06:30
edited on: 2012-05-12 06:32

Justin, Attached is the entire log file from the entire day of Thursday for one of my multi-sim instances that is rarely ever busy.

It is full of the Watchdog timeout errors even though the system was not busy at all.

(0022122)
justincc (administrator)
2012-07-19 15:57

Is this still occurring for you with current git master? What kind of machine are you running on?
(0022123)
argus Portal (reporter)
2012-07-24 13:27

I have this problem too. And when this happens, opensim.exe access very intensive to the harddisk (more than 35 MB/s).

This happens since the last updates (don't remember exactly when it's started)
(0022124)
justincc (administrator)
2012-07-24 16:21
edited on: 2012-07-24 16:21

There are many reasons for these messages and one of them is system contention, in which case your simulator is being delayed by reasons outside of OpenSimulator's control. This may be happening if something else that's causing a lot of hard disk activity is going on.

This is distinct from the extremely small timeout messages reported above which are a bug of some form.

(0022125)
argus Portal (reporter)
2012-07-24 20:02

It is opensim.exe that access to the harddisc. I can see it in ressourcemonitor in win7.
(0022126)
argus Portal (reporter)
2012-07-26 14:50

addition:

Often this happens in conjunction with this message:

[MAP IMAGE SERVICE MODULE]: something bad happened Ung├╝ltiger Parameter.

Translation: "Ung├╝ltiger Parameter" means "invalid parameter"
(0022127)
argus Portal (reporter)
2012-08-12 15:53

Now i am sure: The timeout is caused by the server-software itself: It is the very heavy access to the harddisc.

FIRST the harddisc is very busy because of opensim. And THEN the timeout-messages
fill the console. For every region there comes this block of messages:
______________________________________________________________________________


00:38:23 - [WATCHDOG]: Timeout detected for thread "MapItemRequestThread (altlan
d2)". ThreadState=Background. Last tick was 19063ms ago.
00:38:23 - [WATCHDOG]: Timeout detected for thread "Incoming Packets (altland2)"
. ThreadState=Running. Last tick was 18938ms ago. Data: Client is none
00:38:23 - [WATCHDOG]: Timeout detected for thread "Outgoing Packets (altland2)"
. ThreadState=WaitSleepJoin. Last tick was 16ms ago. Data: Client is none
00:38:23 - [WATCHDOG]: Timeout detected for thread "Heartbeat (altland2)". Threa
dState=WaitSleepJoin. Last tick was 0ms ago.
00:38:23 - [WATCHDOG]: Timeout detected for thread "Maintenance (altland2)". Thr
eadState=WaitSleepJoin. Last tick was 31ms ago.
(0022128)
kenvc (reporter)
2012-08-12 16:12

This still happens a lot if I have more that one instance starting at the same time. If I stagger the startup, it doesnt happen that often during normal operation. I have not noticed the 0ms timeout message lately, so that part may be fixed.
(0022129)
argus Portal (reporter)
2012-08-12 17:00

Here i have just one instance with 10 regions. (The serverstartup works well without problems.)

Some versions ago i don't had this problem. Unfortunately I can't remember the version where it has begun.
(0022130)
justincc (administrator)
2012-08-14 12:46

Yes, if this is happening at startup with heavy regions then it's because the cores are genuinely busy with other tasks. However, this is what one would expect at startup so in the future I may well make a refinement to delay the watchdog until startup has finished if this doesn't hide useful data.

The very low ms report probably occurs because the thread times out, but then it actually does get updated before the text which prints out the error condition. The logger needs to receive a copy of the ThreadWatchdogInfo rather than look at the original to prevent this happening.
(0022131)
justincc (administrator)
2012-08-17 16:00

I made a change to delay starting the thread watchdog until the regions are ready in git master 5a1b6fd. However, I still need to make the change where the reporting data is copied to avoid a race.
(0022132)
justincc (administrator)
2012-08-17 16:56

Okay, the second issue was addressed in git master e4e5237. This should stop these false alarms from occurring.

- Issue History
Date Modified Username Field Change
2012-08-18 04:32 DMX04 New Issue
2012-08-18 04:32 DMX04 Issue generated from: 0006012
2013-05-13 16:01 kenvc Status resolved => closed
2013-05-13 16:01 kenvc Fixed in Version => master (dev code)


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker