[Opensim-users] Help in CPU usage diagnosis, please

[John Hopkin]

I've noticed the past couple of builds I've used have sometimes make
heavy use of the CPU even when idle.  For example, today, after
leaving the system idle overnight, the two virtual CPUs (a single-core
Pentium 4) were running constantly at 96-99% each, whereas normally
I'd expect 35-40% and 12-15% during idle running, most of that
non-Opensim overhead.

When this happens, sometimes - but not always - a reload of the server
fixes it.  I'm only running a single OpenSim.exe, and using OSGrid's
UGAIM.

This is noticeable in-world as lag, and I'd like to find out what's
causing it, but I don't know how to narrow it down, other than by
eliminating various regions and also disabling services such as
scripting and prim physics.  With five regions, and more on the way,
it's pretty difficult to eliminate anything that way, especially with
something that's intermittent like this.

It strikes me that I know very little of what I, as a user, can do to
try to diagnose something like this.  I'd like to have more details
before I file a Mantis, since a description alone can't really be much
use to the devs.  The steps I could take with what knowledge I have
are:

1.  Examine OpenSim.log and look for anything suspicious-looking,
especially if it's happening repeatedly.  Do the same with the
on-screen messages (difficult, since the virtual console only scrolls
back so far).

2.  Look at other vital signs, such as network use and free memory -
in this case, these are normal.  In fact, at 33-38% user memory used
of 2GB for four regions, that's better lately than it's been for a
long time.

3.  Selectively disable various regions and services, as mentioned
above, and monitor the situation over time.  This is difficult, as it
naturally affects our use of the sim in the meantime.

4.  Make sure that it is the OpenSim.exe process that's causing it by
using "ps" (in *nix) and similar tools to examine the process.  (It
is).

What else can I do, in general terms?  What debugging tools and
techniques can a user with little direct experience of OS's internal
machinations use to help pinpoint a problem?  Is there any way of
finding out the resource usage of, say, individual modules?  Of
getting more detailed logs?  Anything?

Please note that the actual problem I'm having at the moment is only
an example - I'm interested in general in being able to be more
proactive in diagnosing Opensim problems, knowing as a developer
myself (though not in this project) how valuable more detail from
users can be in tacking a problem.
-- 
John Hopkin