Show stats
From OpenSimulator
Contents |
Introduction
"show stats" is a new statistics gathering mechanism is slowly being added to OpenSimulator to integrate previously separate and ad-hoc mechanisms. For this system, you can see a short statistics summary with the command "show stats" and a much longer list of statistics with the "show stats all" command (which unfortunately doesn't yet include those in the summary!).
Use the "help show stats" command on the console for more information.
See Monitoring for information on other statistics gathering mechanisms for OpenSimulator.
Statistic types
There are currently three types of statistic.
Number
This is just a number, possibly with a unit. For example
server.threadpool.STPActiveThreads : 15 threads
Change over time
This is a number that also shows the last delta between samples and also a moving average of deltas over the last 20 samples, each taken 5 seconds apart. For example
clientstack.test one.IncomingUDPReceivesCount : 98, 1.2/s, 1.7/s
Percentage
This is a percentage with the antecedent and consequent also given. For example
clientstack.test one.PacketsReused : 37.91% (116/306)
Statistics
This information is highly subject to change as suitable data gathering is an evolving area. Need to keep adding stats.
clientstack
These are stats for the LLUDP clientstack that sends and receives UDP messages for viewers connected with the Second Life protocol. Each stats with clientstack.<region-name>.
| Stat name | Units | Description |
|---|---|---|
| InboxPacketsCount | messages | The number of messages that have been pre-processed by an inbound network thread but are now waiting to be processed by the main Incoming Packets thread. If this number exceeds 0 for more than one sample then your simulator cannot cope with the number of UDP messages that clients are sending. Region performance will be degraded from the user perspective. |
| IncomingPacketPoolCount | messages | The number of packet objects currently in the memory pool to deal with incoming messages. This pool is used to reduce memory churn and hence improve simulator performance. The actual number is never an issue since OpenSimulator will create new pool objects as required. |
| IncomingPacketsMalformedCount | messages | Number of malformed packets that the simulator received from viewers. A high and increasing number may indicate a network attack. |
| IncomingPacketsOrphanedCount | messages | Number of well-formed packets that the simulator received but did not relate to any currently logged in viewer. Because viewers can disconnect in the middle of a datastream, we would expect to see a certain number of such packets in the course of normal operation. However, if the number is very high (e.g. tens of thousands) and/or increasing then this may indicate a network attack. |
httpserver
These are stats for OpenSimulator's built-in HTTP server. Statistics are partitioned (contained) by the incoming port number to which they relate. For example, httpserver.8002.HTTPRequestsServed are all the HTTP requests served that came in to port 8002.
| Stat name | Units | Description |
|---|---|---|
| HTTPRequestsServed | requests | The number of HTTP requests served. |
| ProcessedPollResponses | responses | The number of responses sent to the viewer in response to poll requests. |
| QueuedPollResponses | responses | The number of responses queued for the next viewer poll. |
scene
These are stats relating to the graphical simulated scene.
Each starts with scene.<region-name>.
| Stat name | Units | Description |
|---|---|---|
| RootAgents | avatars | The number of root agents in the scene. A root agent is associated with an avatar since a root agent is needed for full sim functionality (e.g. moving around). |
| ChildAgents | agents | The number of child agents in the scene. A child agent has limited functionality and is used by the viewer mainly to observe events in another scene (i.e. so that the user can see what's occuring in neighbouring regions). |
| FrameTime | ms | The amount of time taken by OpenSimulator to complete the last frame. The standard frame time is 18.18ms. In normal conditions there will be some small variation above and below this number, probably due to the low resolution of thread sleeping. This number may occasionally jump if the underlying VM (Mono or .NET) takes a long time to perform a garbage collect. If this number is consistently higher then your simulator is overloaded. |
| PhysicsTime | ms | Amount of time spent performing physics engine operations in a scene frame. If this is consistently high enough to make the FrameTime exceed the 18.18ms standard then region performance will be degraded. You will need to reduce the number of physical objects in your region. For OpenSimulator 0.7.6, ODE is used single-threaded and so extra cores will make no difference to performance. |
| SpareTime | ms | The amount of time that was left over after OpenSimulator had performed all operations required in the current scene loop. During this period OpenSimulator sleeps the thread. |
| SlowFrames | frames | In theory, this records the number of frames which took more than the standard frame time to execute. However, this measure may be buggy. |
server
These are stats relating to the server itself.
threadpool
These are server threadpool stats.
Each starts with server.threadpool.
| Stat name | Units | Description |
|---|---|---|
| BuiltinThreadpoolIOCPThreadsAvailable | threads | The number of pooled thread objects waiting to handle incoming IO requests. OpenSimulator sets this to a very high number by default. If you ever see this approach zero then your simulator is severely overloaded with respect to processing raw incoming IO requests. |
| BuiltinThreadpoolWorkerThreadsAvailable | threads | The number of pooled thread objects in the Mono/.NET native threadpool available to handle requests. If you have async_call_method = UnsafeQueueUserWorkItem (or QueueUserWorkItem though "unsafe" is fine for our purposes) then is used for all OpenSimulator's threading requirements. If you have async_call_method = SmartThreadPool (which is the current default), then SmartThreadPool handles all the heavy thread launching requirements (and so see the server.threadpool.STP* stats). If this figure approaches zero then you simulator is severely overloaded. |
| STPInUseThreads | threads | The number of SmartThreadPool threads currently in use. If this drops to 0 then the simulator will experience general performance degradation. In this case, try greating increasing the MaxPoolThreads setting in the [Startup] section of OpenSim.ini (e.g. to 150). |
| STPMaxThreads | threads | Maximum number of threads that the SmartThreadPool will use. This is set by the MaxPoolThreads setting in the [Startup] section of OpenSim.ini (or OpenSimDefaults.ini). |
| STPWorkItemsWaiting | items | The number of work items waiting because all SmartThreadPool threads are busy with current work items. You will see this start to climb if STPInUseThreads == STPMaxThreads. You never want to see this situation - if you have work items waiting then try increasing your MaxPoolThreads setting. |
stats record
Please note that in OpenSimulator 0.7.6 and prior to 20th March 2014 in development code, this command was "debug stats record".
This is an experimental facilty starting in OpenSimulator 0.7.6 which will record the results of "show stats all" to a log file every 5 seconds. The aim is to record statistical information throughout a session for later analysis. The logging is enabled with the console command. It works on both the simulator and a ROBUST instance.
# stats record start Now recording all stats to file every 5000ms
To stop logging, either shutdown the server or use the command
# stats record stop Stopped recording stats to file.
The log file used has the same name as the main logfile for the executable with the word "Stats" appended. For instance, turning on stats logging for OpenSim.exe will write to the file OpenSimStats.log. For Robust.exe, this will write to RobustStats.log.
Currently and in OpenSimulator 0.7.6, this logging requires entries in OpenSim.exe.config (and Robust.exe.config for Robust). If you are using your own version of these files then you will need to adapt them for the logging entries.
The performance impact of such logging should be low, though the log file size can grow quite large.
stats save
Command added in development code from 20th March 2014.
This command will save a snapshot of current statistical data to a given path. For example
# stats save stats.txt
If the file already exists, the stats will be appended.
