OpenSimulator is an extremely concurrent application. It uses a large number of threads in
- Persistent threads which are continually used for the same task and not returned to any kind of pooling mechanism (such as scene processing and inbound packet processing).
- Thread pools for shorter-lived processing (e.g. for handling many viewer requests or for processing script events).
- Runtime created threads for inbound processing (e.g. for passing UDP/HTTP data to OpenSimulator code).
- Timer events, such as that triggered to processing statical information to send to viewers (server frames per second, etc).
These are threads that are continually used for the same task and not returned to any kind of pooling mechanism (such as scene processing and inbound packet processing).
The best way to talk about these is in the context of the report generated by the "show threads" command on the simulator console. This information is also logged in the simulator log every 60 minutes. Here is some example output. In this case there are two regions int he simulator,
19 threads are being tracked: ID NAME LAST UPDATE (MS) LIFETIME (MS) PRIORITY STATE 8 PollServiceWorkerThread0 136819 48710999 Lowest WaitSleepJoin 9 PollServiceWorkerThread1 286930 48710998 Lowest WaitSleepJoin 10 PollServiceWorkerThread2 48710998 48710998 Lowest WaitSleepJoin 11 PollServiceWatcherThread 610 48710997 Lowest WaitSleepJoin 13 PollServiceWorkerThread0 135840 48707173 Lowest WaitSleepJoin 14 PollServiceWorkerThread1 287870 48707173 Lowest WaitSleepJoin 15 PollServiceWorkerThread2 48707173 48707173 Lowest WaitSleepJoin 16 PollServiceWatcherThread 450 48707172 Lowest WaitSleepJoin 19 MapItemRequestThread (MyRegion1) 380 48703911 Lowest Background, WaitSleepJoin 21 Incoming Packets (MyRegion1) 50 48681777 Lowest WaitSleepJoin 22 Outgoing Packets (MyRegion1) 0 48681776 Lowest WaitSleepJoin 23 Heartbeat (MyRegion1) 60 48681774 Lowest WaitSleepJoin 27 MapItemRequestThread (MyRegion2) 380 48679575 Lowest Background, WaitSleepJoin 28 Incoming Packets (MyRegion2) 50 48679313 Lowest WaitSleepJoin 29 Outgoing Packets (MyRegion2) 0 48679289 Lowest WaitSleepJoin 30 Heartbeat (MyRegion2) 70 48679288 Lowest WaitSleepJoin 33 Maintenance (MyRegion2) 870 48679183 Lowest WaitSleepJoin 40 AsyncLSLCmdHandlerThread 8 48665246 Lowest Background, WaitSleepJoin 126 Maintenance (MyRegion1) 280 48532481 Lowest WaitSleepJoin
Firstly, some general notes about the columns.
- ID - The runtime ID for this particular thread.
- NAME - The name of this persistent thread as assigned by OpenSimulator.
- LAST UPDATE (MS) - The last time that this thread told the thread watchdog that it is still alive. Threads do this periodically so that we can do crude detection of deadlocks and other issues. All threads apart from the PollServiceWorkerThreads will trigger an alarm if they don't check in within 5 seconds.
- LIFETIME (MS) - The number of milliseconds for which a thread has been running. Usually, this will be very close to the time for which the simulator itself has been running, since persistent threads are only currently stopped and started in debugging situations.
- PRIORITY - Thread priority relevant to other threads. This is not implemented in Mono so OpenSimulator tends to leave all threads at the Lowest priority.
- STATE - Thread state, whether the thread is stopped, sleeping/waiting, aborted, etc. As of Mono 2.10.9, it's possible that the Running state is not implemented, leaving all scripts in the WaitSleepJoin state even if they are actually Running.
Secondly, information on these different threads.
- PollServiceWorkerThreadN - A thread used by viewer event queue processing to process events given to them by the PollServiceWatcherThread. There are currently 3 such threads per region. As stated above, these never check in and never trigger a timeout alarm, unlike all other persistent threads.
- PollServiceWatcherThread - This takes event queue requests received from viewers via OpenSimulator's bundled C# WebServer and distributes them to PollServiceWatcherThreads. Unlike PollServiceWorkerThreads, this does check in
- MapItemRequestThread (<region-name>) - Requests map items from other regions. One per region.
- Incoming Packets (<region-name>) - Handles queued incoming LLUDP protocol packets. One per region.
- Outcoming Packets (<region-name>) - Sends queued outgoing LLUDP protocol packets to viewers. One per region.
- AsyncLSLCmdHandlerThread - Performs asynchronous processing related to various scripting facilities. This currently covers dataserver requests, timer firing, HTTP operations, listeners, sensors and XMLRPC requests. Currently, one per simulator.
General thread pool
By default, OpenSimulator uses a library known as the SmartThreadPool for thread pooling. One instance of this handles general server operations, such as viewer request processing. This is used across all regions hosted by that simulator.
As with persistent threads, you can see this information using the "show threads" command on the simulator console.
Here is some example output.
Main threadpool (excluding script engine pools) Thread pool used : SmartThreadPool Max threads : 150 Min threads : 2 Allocated threads : 2 In use threads : 0 Work items waiting : 0
- Max threads - The maximum number of threads that will be used by this pool. If this limit is reached then further requests will have to wait for a thread to be returned to the pool.
- Min threads - The minimum number of threads that this pool will keep on hand even if no work is being performed.
- Allocated threads - The number of threads currently in the pool.
- In use threads - The number of threads currently performing tasks.
- Work items waiting - The number of work items waiting for another thread pool thread to become available.
XEngine thread pools
Each XEngine script engine will use an additional SmartThreadPool instance for its own internal operations, such as script event processing. As there is one XEngine script engine per region, this will also mean an additional thread pool per region.
Status of xengine thread pools can be seen by using the "xengine status" command on the simulator console. Here is some example output.
Status of XEngine instance for MyRegion1 Scripts loaded : 2491 Unique scripts : 842 Scripts waiting for load : 0 Max threads : 100 Min threads : 2 Allocated threads : 36 In use threads : 18 Work items waiting : 0 Sensors : 11 Dataserver requests : 0 Timers : 36 Listeners : 59 Status of XEngine instance for MyRegion2 Scripts loaded : 0 Unique scripts : 0 Scripts waiting for load : 0 Max threads : 100 Min threads : 2 Allocated threads : 3 In use threads : 2 Work items waiting : 0 Sensors : 0 Dataserver requests : 0 Timers : 0 Listeners : 0
In this case there are two regions, MyRegion1 and MyRegion2, and hence two thread pools. The thread pool details are the lines which match the data given for the general thread pool (Unique scripts, allocated threads, etc.). In this case, the "xengine status" command also shows additional details (scripts loaded, sensors, etc.) which are not directly thread related.