Revision as of 13:17, 22 October 2020

Introduction

OpenSimulator is an extremely concurrent application. It uses a large number of threads

1. Persistent threads. Threads which are continually used for the same task and not returned to any kind of pooling mechanism.
2. Thread pools.
3. Jobs. A job is like a pool with a single thread.
4. Timer events or IO threads.
5. Tasks.

Persistent threads

These are threads that are continually used for the same task and not returned to any kind of pooling mechanism (such as scene processing and inbound packet processing).

The best way to talk about these is in the context of the report generated by the "show threads" command on the simulator console. This information is also logged in the simulator log every 60 minutes. Here is some example output. In this case there are two regions int he simulator,

19 threads are being tracked:
    ID                                  NAME   LAST UPDATE (MS)   LIFETIME (MS)     PRIORITY                            STATE
     8              PollServiceWorkerThread0             136819        48710999       Lowest                    WaitSleepJoin
     9              PollServiceWorkerThread1             286930        48710998       Lowest                    WaitSleepJoin
    10              PollServiceWorkerThread2           48710998        48710998       Lowest                    WaitSleepJoin
    11              PollServiceWatcherThread                610        48710997       Lowest                    WaitSleepJoin
    13              PollServiceWorkerThread0             135840        48707173       Lowest                    WaitSleepJoin
    14              PollServiceWorkerThread1             287870        48707173       Lowest                    WaitSleepJoin
    15              PollServiceWorkerThread2           48707173        48707173       Lowest                    WaitSleepJoin
    16              PollServiceWatcherThread                450        48707172       Lowest                    WaitSleepJoin
    19      MapItemRequestThread (MyRegion1)                380        48703911       Lowest        Background, WaitSleepJoin
    21          Incoming Packets (MyRegion1)                 50        48681777       Lowest                    WaitSleepJoin
    22          Outgoing Packets (MyRegion1)                  0        48681776       Lowest                    WaitSleepJoin
    23                 Heartbeat (MyRegion1)                 60        48681774       Lowest                    WaitSleepJoin
    27      MapItemRequestThread (MyRegion2)                380        48679575       Lowest        Background, WaitSleepJoin
    28          Incoming Packets (MyRegion2)                 50        48679313       Lowest                    WaitSleepJoin
    29          Outgoing Packets (MyRegion2)                  0        48679289       Lowest                    WaitSleepJoin
    30                 Heartbeat (MyRegion2)                 70        48679288       Lowest                    WaitSleepJoin
    33               Maintenance (MyRegion2)                870        48679183       Lowest                    WaitSleepJoin
    40              AsyncLSLCmdHandlerThread                  8        48665246       Lowest        Background, WaitSleepJoin
   126               Maintenance (MyRegion1)                280        48532481       Lowest                    WaitSleepJoin

Firstly, some general notes about the columns.

• ID - The runtime ID for this particular thread.
• NAME - The name of this persistent thread as assigned by OpenSimulator.
• LAST UPDATE (MS) - The last time that this thread told the thread watchdog that it is still alive. Threads do this periodically so that we can do crude detection of deadlocks and other issues. All threads apart from the PollServiceWorkerThreads will trigger an alarm if they don't check in within 5 seconds.
• LIFETIME (MS) - The number of milliseconds for which a thread has been running. Usually, this will be very close to the time for which the simulator itself has been running, since persistent threads are only currently stopped and started in debugging situations.
• PRIORITY - Thread priority relevant to other threads. This is not implemented in Mono so OpenSimulator tends to leave all threads at the Lowest priority.
• STATE - Thread state, whether the thread is stopped, sleeping/waiting, aborted, etc. As of Mono 2.10.9, it's possible that the Running state is not implemented, leaving all scripts in the WaitSleepJoin state even if they are actually Running.

Secondly, information on these different threads.

• PollServiceWorkerThreadN - A thread used by viewer event queue processing to process events given to them by the PollServiceWatcherThread. There are currently 3 such threads per region. As stated above, these never check in and never trigger a timeout alarm, unlike all other persistent threads.
• PollServiceWatcherThread - This takes event queue requests received from viewers via OpenSimulator's bundled C# WebServer and distributes them to PollServiceWatcherThreads. Unlike PollServiceWorkerThreads, this does check in
• MapItemRequestThread (<region-name>) - Requests map items from other regions. One per region.
• Incoming Packets (<region-name>) - Handles queued incoming LLUDP protocol packets. One per region.
• Outcoming Packets (<region-name>) - Sends queued outgoing LLUDP protocol packets to viewers. One per region.
• AsyncLSLCmdHandlerThread - Performs asynchronous processing related to various scripting facilities. This currently covers dataserver requests, timer firing, HTTP operations, listeners, sensors and XMLRPC requests. Currently, one per simulator.

Note: Due to several practical issues, not all threads are listed

Thread pools

A thread is a expensive program resource that takes a lot of time to create and release. A thread pool allows a reduction of the number of suck operations, reusing a thread while it is needed.
Instead of releasing a thread once a job is done, the thread is kept on a pool for sometime, and only released if was not needed during that time
The pool may keep a minimum number of threads, so a new work item does not need to wait for the time it takes to create a new thread.
When a thread starts executing a job a new thread may also be created in parallel, if the total number of threads of the pool is less than a maximum, to ensure there are available threads for possible job requests that may happen during the execution time.

General thread pool

OpenSimulator has a general thread pool that can be configured to use the .net main thread pool or one provided by the library SmartThreadPool
This handles general server operations, such as viewer request processing. This is used across all regions hosted by that simulator.

As with persistent threads, you can see this information using the "show threads" command on the simulator console.

Here is some example output.

Main threadpool (excluding script engine pools)
Thread pool used           : SmartThreadPool
Max threads                : 150
Min threads                : 2
Allocated threads          : 2
In use threads             : 0
Work items waiting         : 0

• Max threads - The maximum number of threads that will be used by this pool. If this limit is reached then further requests will have to wait for a thread to be returned to the pool.
• Min threads - The minimum number of threads that this pool will keep on hand even if no work is being performed.
• Allocated threads - The number of threads currently in the pool.
• In use threads - The number of threads currently performing tasks.
• Work items waiting - The number of work items waiting for another thread pool thread to become available.

XEngine thread pools

Each XEngine script engine will use an additional SmartThreadPool instance for its own internal operations, such as script event processing. As there is one XEngine script engine per region, this will also mean an additional thread pool per region.

Status of xengine thread pools can be seen by using the "xengine status" command on the simulator console. Here is some example output.

Status of XEngine instance for MyRegion1
Scripts loaded             : 2491
Unique scripts             : 842
Scripts waiting for load   : 0
Max threads                : 100
Min threads                : 2
Allocated threads          : 36
In use threads             : 18
Work items waiting         : 0
Sensors                    : 11
Dataserver requests        : 0
Timers                     : 36
Listeners                  : 59

Status of XEngine instance for MyRegion2
Scripts loaded             : 0
Unique scripts             : 0
Scripts waiting for load   : 0
Max threads                : 100
Min threads                : 2
Allocated threads          : 3
In use threads             : 2
Work items waiting         : 0
Sensors                    : 0
Dataserver requests        : 0
Timers                     : 0
Listeners                  : 0

In this case there are two regions, MyRegion1 and MyRegion2, and hence two thread pools. The thread pool details are the lines which match the data given for the general thread pool (Unique scripts, allocated threads, etc.). In this case, the "xengine status" command also shows additional details (scripts loaded, sensors, etc.) which are not directly thread related.

Other thread pools

Additional SmartThreadPool instance used on several modules.

Timers IO threads

This are threads from .Net main Threadpool.
Either to execute a timer event code or the final stage of a IO operation.
A task can look and behave as a thread, but may not have one permanently assigned to it, That is managed by the runtime.
Tasks may also be used as alternative to the use of a pool thread for short work items, or perform IO, in several parts of code.

Tasks

A task can look and behave as a thread, but may not have one permanently assigned to it, That is managed by the runtime.
Tasks may also be used as alternative to the use of a pool thread for short work items, or perform IO, in several parts of code.