Show stats/de
From OpenSimulator
 |
Languages: |
|
Introduction
"show stats" is a new statistics gathering mechanism is slowly being added to OpenSimulator to integrate previously separate and ad-hoc mechanisms. For this system, you can see a short statistics summary with the command "show stats" and a much longer list of statistics with the "show stats all" command (which unfortunately doesn't yet include those in the summary!).
Use the "help show stats" command on the console for more information.
See Monitoring for information on other statistics gathering mechanisms for OpenSimulator.
See Stats Manager for information on how to register stats with this mechanism.
General Concepts
The full name of a stat has three components, separated by periods (.). In order, these are
- category - The category of the stat (e.g. server, scene)
- container - The containing object for this stat. For example, on scene this is the scene name (e.g. region 1). For server, this is the component to which the stat relates (e.g. memory, processor).
- name - The stat itself. For example, ClientLogoutsDueToNoReceives or SimFPS.
Console commands
From the console, there are several ways to display the stats:
- show stats returns simulator statistics in a legacy format
- show stats all displays all the registered stats one line per stat.
- show stats category container displays all the stats within a particular category/container
Recording
Saving statistics information can be very useful for later analysis. There are commands to save both a snapshot and a continuous record of stats information. There is also a facility to fetch stats information via a URL.
stats save console command
Command added in development code from 20th March 2014.
This command will save a snapshot of current statistical data to a given path. For example
# stats save stats.txt
If the file already exists, the stats will be appended.
stats record console command
Please note that in OpenSimulator 0.7.6 this command was "debug stats record".
Unfortunately, a bug in OpenSimulator 0.8 means that the command "stats record start|stop" will not work properly and will report a usage error with the old command name. The workaround is to repeat the last command twice (e.g. "stats record start start").
This bug was fixed in OpenSimulator 0.8.0.1
This is an experimental facilty starting in OpenSimulator 0.7.6 which will record the results of "show stats all" to a log file every 5 seconds. The aim is to record statistical information throughout a session for later analysis. The logging is enabled with the console command. It works on both the simulator and a ROBUST instance.
# stats record start Now recording all stats to file every 5000ms
To stop logging, either shutdown the server or use the command
# stats record stop Stopped recording stats to file.
The log file used has the same name as the main logfile for the executable with the word "Stats" appended. For instance, turning on stats logging for OpenSim.exe will write to the file OpenSimStats.log. For Robust.exe, this will write to RobustStats.log.
Currently and in OpenSimulator 0.7.6, this logging requires entries in OpenSim.exe.config (and Robust.exe.config for Robust). If you are using your own version of these files then you will need to adapt them for the logging entries.
The performance impact of such logging should be low, though the log file size can grow quite large.
Web access to stats
If [Startup]ManagedStatsRemoteFetchURI is defined and set to some value, the simulator statistics are available over the web via HTTP GET operations. The URL is the simulator's HTTP server URL appended with the value in ManagedStatsRemoteFetchURI.
The URL fetched accepts four different query parameters:
- cat which specifies a category to fetch. If no category is given, the special category "all" is use which return all categories;
- cont specified a container. If not specified, the special container "all" is used to return all containers
- stat specifies the name of the particular stat. If not specified, the stat "all" is used to return all stats.
- callback specifies if the returned JSON data is to be wrapped in a callback function.
For instance, for a simulator at IP address 10.1.1.4 with the INI settings:
[Network] http_listener_port=9000 [Startup] ManagedStatsRemoteFetchURI=ManagedStats
example data fetches are:
- wget http://10.1.1.4:9000/ManagedStats/
- returns all stats registered in the simulator
- wget http://10.1.1.4:9000/ManagedStats/?cat=server
- returns all stats registered with the category "server"
- wget http://10.1.1.4:9000/ManagedStats/?cat=server&cont=threadpool
- returns all stats for the "threadpool" container within the "server" category
- wget http://10.1.1.4:9000/ManagedStats/?cat=server&cont=threadpool&stat=STPMinThreads
- returns the value for the specific stat
The data is returned in JSON format grouped as you might expect with maps of categories containing maps of containers containing maps of stats. For instance, the response to http://10.1.1.4:9000/ManagedStats/?cat=server&cont=threadpool might look like:
{"server":
{"threadpool":
{"BuiltinThreadpoolIOCPThreadsAvailable":
{"StatType":"Stat",
"Category":"server",
"Container":"threadpool",
"ShortName":"BuiltinThreadpoolIOCPThreadsAvailable",
"Name":"BuiltinThreadpoolIOCPThreadsAvailable",
"Description":"BuiltinThreadpoolIOCPThreadsAvailable",
"UnitName":"threads",
"Value":1000.0
},
"BuiltinThreadpoolWorkerThreadsAvailable":
{"StatType":"Stat",
"Category":"server",
"Container":"threadpool",
"ShortName":"BuiltinThreadpoolWorkerThreadsAvailable",
"Name":"BuiltinThreadpoolWorkerThreadsAvailable",
"Description":"BuiltinThreadpoolWorkerThreadsAvailable",
"UnitName":"threads",
"Value":499.0
},
"STPActiveThreads":
{"StatType":"Stat",
"Category":"server",
"Container":"threadpool",
"ShortName":"STPActiveThreads",
"Name":"STPActiveThreads",
"Description":"STPActiveThreads",
"UnitName":"threads",
"Value":15.0
},
"STPConcurrency":
{"StatType":"Stat",
"Category":"server",
"Container":"threadpool",
"ShortName":"STPConcurrency",
"Name":"STPConcurrency",
"Description":"STPConcurrency",
"UnitName":"threads",
"Value":45.0
},
"STPInUseThreads":
{"StatType":"Stat",
"Category":"server",
"Container":"threadpool",
"ShortName":"STPInUseThreads",
"Name":"STPInUseThreads",
"Description":"STPInUseThreads",
"UnitName":"threads",
"Value":0.0
},
"STPMaxThreads":
{"StatType":"Stat",
"Category":"server",
"Container":"threadpool",
"ShortName":"STPMaxThreads",
"Name":"STPMaxThreads",
"Description":"STPMaxThreads",
"UnitName":"threads",
"Value":45.0
},
"STPMinThreads":
{"StatType":"Stat",
"Category":"server",
"Container":"threadpool",
"ShortName":"STPMinThreads",
"Name":"STPMinThreads",
"Description":"STPMinThreads",
"UnitName":"threads",
"Value":15.0
},
"STPWorkItemsWaiting":
{"StatType":"Stat",
"Category":"server",
"Container":"threadpool",
"ShortName":"STPWorkItemsWaiting",
"Name":"STPWorkItemsWaiting",
"Description":"STPWorkItemsWaiting",
"UnitName":"threads",
"Value":0.0
}
}
}
}
Analysis
Tools to analyze record stats data are under development at https://github.com/justincc/opensimulator-tools/tree/master/analysis/opensimulator-stats-analyzer. This includes summary and graphing tools.
Types
There are currently three types of statistic.
Number
This is just a number, possibly with a unit. For example
server.threadpool.STPActiveThreads : 15 threads
Change over time
This is a number that also shows the last delta between samples and also a moving average of deltas over the last 20 samples, each taken 5 seconds apart. For example
clientstack.test one.IncomingUDPReceivesCount : 98, 1.2/s, 1.7/s
Percentage
This is a percentage with the antecedent and consequent also given. For example
clientstack.test one.PacketsReused : 37.91% (116/306)
Statistics
This information is highly subject to change as suitable data gathering is an evolving area. Need to keep adding stats.
clientstack
These are stats for the LLUDP clientstack that sends and receives UDP messages for viewers connected with the Second Life protocol. Each stats with clientstack.<region-name>.
| Stat name | Units | Description |
|---|---|---|
| InboxPacketsCount | messages | The number of messages that have been pre-processed by an inbound network thread but are now waiting to be processed by the main Incoming Packets thread. If this number exceeds 0 for more than one sample then your simulator cannot cope with the number of UDP messages that clients are sending. Region performance will be degraded from the user perspective. |
| IncomingPacketPoolCount | messages | The number of packet objects currently in the memory pool to deal with incoming messages. This pool is used to reduce memory churn and hence improve simulator performance. The actual number is never an issue since OpenSimulator will create new pool objects as required. |
| IncomingPacketsMalformedCount | messages | Number of malformed packets that the simulator received from viewers. A high and increasing number may indicate a network attack. |
| IncomingPacketsOrphanedCount | messages | Number of well-formed packets that the simulator received but did not relate to any currently logged in viewer. Because viewers can disconnect in the middle of a datastream, we would expect to see a certain number of such packets in the course of normal operation. However, if the number is very high (e.g. tens of thousands) and/or increasing then this may indicate a network attack. |
httpserver
These are stats for OpenSimulator's built-in HTTP server. Statistics are partitioned (contained) by the incoming port number to which they relate. For example, httpserver.8002.HTTPRequestsServed are all the HTTP requests served that came in to port 8002.
| Stat name | Units | Description |
|---|---|---|
| HTTPRequestsServed | requests | The number of HTTP requests served. |
| ProcessedPollResponses | responses | The number of responses sent to the viewer in response to poll requests. |
| QueuedPollResponses | responses | The number of responses queued for the next viewer poll. |
scene
These are stats relating to the graphical simulated scene.
Each starts with scene.<region-name>.
| Stat name | Units | Description |
|---|---|---|
| RootAgents | avatars | The number of root agents in the scene. A root agent is associated with an avatar since a root agent is needed for full sim functionality (e.g. moving around). |
| ChildAgents | agents | The number of child agents in the scene. A child agent has limited functionality and is used by the viewer mainly to observe events in another scene (i.e. so that the user can see what's occuring in neighbouring regions). |
| FrameTime | ms | The amount of time taken by OpenSimulator to complete the last frame. The standard frame time is 18.18ms. In normal conditions there will be some small variation above and below this number, probably due to the low resolution of thread sleeping. This number may occasionally jump if the underlying VM (Mono or .NET) takes a long time to perform a garbage collect. If this number is consistently higher then your simulator is overloaded. |
| PhysicsTime | ms | Amount of time spent performing physics engine operations in a scene frame. If this is consistently high enough to make the FrameTime exceed the 18.18ms standard then region performance will be degraded. You will need to reduce the number of physical objects in your region. For OpenSimulator 0.7.6, ODE is used single-threaded and so extra cores will make no difference to performance. |
| SpareTime | ms | The amount of time that was left over after OpenSimulator had performed all operations required in the current scene loop. During this period OpenSimulator sleeps the thread. |
| SlowFrames | frames | In theory, this records the number of frames which took more than the standard frame time to execute. However, this measure may be buggy. |
server
These are stats relating to the server itself.
thread
These are the times in milliseconds since the monitored thread last performed any activity. This is the same as the "LAST UDPATE" information shown by the "show threads" command.
threadpool
These are server threadpool stats.
Each starts with server.threadpool.
| Stat name | Units | Description |
|---|---|---|
| BuiltinThreadpoolIOCPThreadsAvailable | threads | The number of pooled thread objects waiting to handle incoming IO requests. OpenSimulator sets this to a very high number by default. If you ever see this approach zero then your simulator is severely overloaded with respect to processing raw incoming IO requests. |
| BuiltinThreadpoolWorkerThreadsAvailable | threads | The number of pooled thread objects in the Mono/.NET native threadpool available to handle requests. If you have async_call_method = UnsafeQueueUserWorkItem (or QueueUserWorkItem though "unsafe" is fine for our purposes) then is used for all OpenSimulator's threading requirements. If you have async_call_method = SmartThreadPool (which is the current default), then SmartThreadPool handles all the heavy thread launching requirements (and so see the server.threadpool.STP* stats). If this figure approaches zero then you simulator is severely overloaded. |
| STPInUseThreads | threads | The number of SmartThreadPool threads currently in use. If this climbs to the maximum allowed by STPMaxThreads then the simulator will experience general performance degradation. In this case, try increasing the MaxPoolThreads setting in the [Startup] section of OpenSim.ini (e.g. to 150). |
| STPMaxThreads | threads | Maximum number of threads that the SmartThreadPool will use. This is set by the MaxPoolThreads setting in the [Startup] section of OpenSim.ini (or OpenSimDefaults.ini). |
| STPWorkItemsWaiting | items | The number of work items waiting because all SmartThreadPool threads are busy with current work items. You will see this start to climb if STPInUseThreads == STPMaxThreads. You never want to see this situation - if you have work items waiting then try increasing your MaxPoolThreads setting. |
opensimulator.org website hosting kindly provided by
