Threading

=Introduction=

OpenSimulator is an extremely concurrent application. It uses a large number of threads in


 * 1) Persistent threads which are continually used for the same task and not returned to any kind of pooling mechanism (such as scene processing and inbound packet processing).
 * 2) Thread pools for shorter-lived processing (e.g. for handling many viewer requests or for processing script events).
 * 3) Runtime created threads for inbound processing (e.g. for passing UDP/HTTP data to OpenSimulator code).
 * 4) Timer events, such as that triggered to processing statical information to send to viewers (server frames per second, etc).

=Persistent threads=

These are threads that are continually used for the same task and not returned to any kind of pooling mechanism (such as scene processing and inbound packet processing).

The best way to talk about these is in the context of the report generated by the "show threads" command on the simulator console. This information is also logged in the simulator log every 60 minutes. Here is some example output. In this case there are two regions int he simulator,

19 threads are being tracked: ID                                 NAME   LAST UPDATE (MS)   LIFETIME (MS)     PRIORITY                            STATE 8             PollServiceWorkerThread0             136819        48710999       Lowest                    WaitSleepJoin 9             PollServiceWorkerThread1             286930        48710998       Lowest                    WaitSleepJoin 10             PollServiceWorkerThread2           48710998        48710998       Lowest                    WaitSleepJoin 11             PollServiceWatcherThread                610        48710997       Lowest                    WaitSleepJoin 13             PollServiceWorkerThread0             135840        48707173       Lowest                    WaitSleepJoin 14             PollServiceWorkerThread1             287870        48707173       Lowest                    WaitSleepJoin 15             PollServiceWorkerThread2           48707173        48707173       Lowest                    WaitSleepJoin 16             PollServiceWatcherThread                450        48707172       Lowest                    WaitSleepJoin 19     MapItemRequestThread (MyRegion1)                380        48703911       Lowest        Background, WaitSleepJoin 21         Incoming Packets (MyRegion1)                 50        48681777       Lowest                    WaitSleepJoin 22         Outgoing Packets (MyRegion1)                  0        48681776       Lowest                    WaitSleepJoin 23                Heartbeat (MyRegion1)                 60        48681774       Lowest                    WaitSleepJoin 27     MapItemRequestThread (MyRegion2)                380        48679575       Lowest        Background, WaitSleepJoin 28         Incoming Packets (MyRegion2)                 50        48679313       Lowest                    WaitSleepJoin 29         Outgoing Packets (MyRegion2)                  0        48679289       Lowest                    WaitSleepJoin 30                Heartbeat (MyRegion2)                 70        48679288       Lowest                    WaitSleepJoin 33              Maintenance (MyRegion2)                870        48679183       Lowest                    WaitSleepJoin 40             AsyncLSLCmdHandlerThread                  8        48665246       Lowest        Background, WaitSleepJoin 126              Maintenance (MyRegion1)                280        48532481       Lowest                    WaitSleepJoin

Firstly, some general notes about the columns.


 * ID - The runtime ID for this particular thread.
 * NAME - The name of this persistent thread as assigned by OpenSimulator.
 * LAST UPDATE (MS) - The last time that this thread told the thread watchdog that it is still alive. Threads do this periodically so that we can do crude detection of deadlocks and other issues.  All threads apart from the PollServiceWorkerThreads will trigger an alarm if they don't check in within 5 seconds.
 * LIFETIME (MS) - The number of milliseconds for which a thread has been running. Usually, this will be very close to the time for which the simulator itself has been running, since persistent threads are only currently stopped and started in debugging situations.
 * PRIORITY - Thread priority relevant to other threads. This is not implemented in Mono so OpenSimulator tends to leave all threads at the Lowest priority.
 * STATE - Thread state, whether the thread is stopped, sleeping/waiting, aborted, etc. As of Mono 2.10.9, it's possible that the Running state is not implemented, leaving all scripts in the WaitSleepJoin state even if they are actually Running.

Secondly, information on these different threads.


 * PollServiceWorkerThreadN - A thread used by viewer event queue processing to process events given to them by the PollServiceWatcherThread. There are currently 3 such threads per region.  As stated above, these never check in and never trigger a timeout alarm, unlike all other persistent threads.
 * PollServiceWatcherThread - This takes event queue requests received from viewers via OpenSimulator's bundled C# WebServer and distributes them to PollServiceWatcherThreads. Unlike PollServiceWorkerThreads, this does check in
 * MapItemRequestThread () - Requests map items from other regions. One per region.
 * Incoming Packets () - Handles queued incoming LLUDP protocol packets. One per region.
 * Outcoming Packets () - Sends queued outgoing LLUDP protocol packets to viewers. One per region.
 * AsyncLSLCmdHandlerThread - Performs asynchronous processing related to various scripting facilities. This currently covers dataserver requests, timer firing, HTTP operations, listeners, sensors and XMLRPC requests.  Currently, one per simulator.

=Thread pools=

General thread pool
By default, OpenSimulator uses a library known as the SmartThreadPool for thread pooling. One instance of this handles general server operations, such as viewer request processing. This is used across all regions hosted by that simulator.

As with persistent threads, you can see this information using the "show threads" command on the simulator console.

Here is some example output.

Main threadpool (excluding script engine pools) Thread pool used          : SmartThreadPool Max threads               : 150 Min threads               : 2 Allocated threads         : 2 In use threads            : 0 Work items waiting        : 0


 * Max threads - The maximum number of threads that will be used by this pool. If this limit is reached then further requests will have to wait for a thread to be returned to the pool.
 * Min threads - The minimum number of threads that this pool will keep on hand even if no work is being performed.
 * Allocated threads - The number of threads currently in the pool.
 * In use threads - The number of threads currently performing tasks.
 * Work items waiting - The number of work items waiting for another thread pool thread to become available.

XEngine thread pools
Each XEngine script engine will use an additional SmartThreadPool instance for its own internal operations, such as script event processing. As there is one XEngine script engine per region, this will also mean an additional thread pool per region.

Status of xengine thread pools can be seen by using the "xengine status" command on the simulator console. Here is some example output.

Status of XEngine instance for MyRegion1 Scripts loaded            : 2491 Unique scripts            : 842 Scripts waiting for load  : 0 Max threads               : 100 Min threads               : 2 Allocated threads         : 36 In use threads            : 18 Work items waiting        : 0 Sensors                   : 11 Dataserver requests       : 0 Timers                    : 36 Listeners                 : 59

Status of XEngine instance for MyRegion2 Scripts loaded            : 0 Unique scripts            : 0 Scripts waiting for load  : 0 Max threads               : 100 Min threads               : 2 Allocated threads         : 3 In use threads            : 2 Work items waiting        : 0 Sensors                   : 0 Dataserver requests       : 0 Timers                    : 0 Listeners                 : 0

In this case there are two regions, MyRegion1 and MyRegion2, and hence two thread pools. The thread pool details are the lines which match the data given for the general thread pool (Unique scripts, allocated threads, etc.). In this case, the "xengine status" command also shows additional details (scripts loaded, sensors, etc.) which are not directly thread related.