Threading
From OpenSimulator
Contents |
Introduction
OpenSimulator is an extremely concurrent application. It uses a large number of threads
- Persistent threads. Threads which are continually used for the same task and not returned to any kind of pooling mechanism.
- Thread pools.
- Jobs. A job is like a pool with a single thread.
- Timer events.
- Tasks or continuation threads.
Persistent threads
These are threads that are continually used for the same task and not returned to any kind of pooling mechanism (such as scene processing and inbound packet processing).
The best way to talk about these is in the context of the report generated by the "show threads" command on the simulator console. This information is also logged in the simulator log every 60 minutes. Here is some example output. In this case there are two regions int he simulator,
19 threads are being tracked:
ID NAME LAST UPDATE (MS) LIFETIME (MS) PRIORITY STATE
8 PollServiceWorkerThread0 136819 48710999 Lowest WaitSleepJoin
9 PollServiceWorkerThread1 286930 48710998 Lowest WaitSleepJoin
10 PollServiceWorkerThread2 48710998 48710998 Lowest WaitSleepJoin
11 PollServiceWatcherThread 610 48710997 Lowest WaitSleepJoin
13 PollServiceWorkerThread0 135840 48707173 Lowest WaitSleepJoin
14 PollServiceWorkerThread1 287870 48707173 Lowest WaitSleepJoin
15 PollServiceWorkerThread2 48707173 48707173 Lowest WaitSleepJoin
16 PollServiceWatcherThread 450 48707172 Lowest WaitSleepJoin
19 MapItemRequestThread (MyRegion1) 380 48703911 Lowest Background, WaitSleepJoin
21 Incoming Packets (MyRegion1) 50 48681777 Lowest WaitSleepJoin
22 Outgoing Packets (MyRegion1) 0 48681776 Lowest WaitSleepJoin
23 Heartbeat (MyRegion1) 60 48681774 Lowest WaitSleepJoin
27 MapItemRequestThread (MyRegion2) 380 48679575 Lowest Background, WaitSleepJoin
28 Incoming Packets (MyRegion2) 50 48679313 Lowest WaitSleepJoin
29 Outgoing Packets (MyRegion2) 0 48679289 Lowest WaitSleepJoin
30 Heartbeat (MyRegion2) 70 48679288 Lowest WaitSleepJoin
33 Maintenance (MyRegion2) 870 48679183 Lowest WaitSleepJoin
40 AsyncLSLCmdHandlerThread 8 48665246 Lowest Background, WaitSleepJoin
126 Maintenance (MyRegion1) 280 48532481 Lowest WaitSleepJoin
Firstly, some general notes about the columns.
- ID - The runtime ID for this particular thread.
- NAME - The name of this persistent thread as assigned by OpenSimulator.
- LAST UPDATE (MS) - The last time that this thread told the thread watchdog that it is still alive. Threads do this periodically so that we can do crude detection of deadlocks and other issues. All threads apart from the PollServiceWorkerThreads will trigger an alarm if they don't check in within 5 seconds.
- LIFETIME (MS) - The number of milliseconds for which a thread has been running. Usually, this will be very close to the time for which the simulator itself has been running, since persistent threads are only currently stopped and started in debugging situations.
- PRIORITY - Thread priority relevant to other threads. This is not implemented in Mono so OpenSimulator tends to leave all threads at the Lowest priority.
- STATE - Thread state, whether the thread is stopped, sleeping/waiting, aborted, etc. As of Mono 2.10.9, it's possible that the Running state is not implemented, leaving all scripts in the WaitSleepJoin state even if they are actually Running.
Secondly, information on these different threads.
- PollServiceWorkerThreadN - A thread used by viewer event queue processing to process events given to them by the PollServiceWatcherThread. There are currently 3 such threads per region. As stated above, these never check in and never trigger a timeout alarm, unlike all other persistent threads.
- PollServiceWatcherThread - This takes event queue requests received from viewers via OpenSimulator's bundled C# WebServer and distributes them to PollServiceWatcherThreads. Unlike PollServiceWorkerThreads, this does check in
- MapItemRequestThread (<region-name>) - Requests map items from other regions. One per region.
- Incoming Packets (<region-name>) - Handles queued incoming LLUDP protocol packets. One per region.
- Outcoming Packets (<region-name>) - Sends queued outgoing LLUDP protocol packets to viewers. One per region.
- AsyncLSLCmdHandlerThread - Performs asynchronous processing related to various scripting facilities. This currently covers dataserver requests, timer firing, HTTP operations, listeners, sensors and XMLRPC requests. Currently, one per simulator.
Thread pools
A thread is a expensive program resource that takes a lot of time to create and release. A thread pool allows a reduction of the number of suck operations, reusing a thread while it is needed.
Instead of releasing a thread once a job is done, the thread is kept on a pool for sometime, and only released if was not needed during that time
The pool may keep a minimum number of threads, so a new work item does not need to wait for the time it takes to create a new thread.
When a thread starts executing a job a new thread may also be created in parallel, if the total number of threads of the pool is less than a maximum, to ensure there are available threads for possible job requests that may happen during the execution time.
General thread pool
OpenSimulator has a general thread pool that can be configured to use the .net main thread pool or one provided by the library SmartThreadPool
This handles general server operations, such as viewer request processing. This is used across all regions hosted by that simulator.
As with persistent threads, you can see this information using the "show threads" command on the simulator console.
Here is some example output.
Main threadpool (excluding script engine pools) Thread pool used : SmartThreadPool Max threads : 150 Min threads : 2 Allocated threads : 2 In use threads : 0 Work items waiting : 0
- Max threads - The maximum number of threads that will be used by this pool. If this limit is reached then further requests will have to wait for a thread to be returned to the pool.
- Min threads - The minimum number of threads that this pool will keep on hand even if no work is being performed.
- Allocated threads - The number of threads currently in the pool.
- In use threads - The number of threads currently performing tasks.
- Work items waiting - The number of work items waiting for another thread pool thread to become available.
XEngine thread pools
Each XEngine script engine will use an additional SmartThreadPool instance for its own internal operations, such as script event processing. As there is one XEngine script engine per region, this will also mean an additional thread pool per region.
Status of xengine thread pools can be seen by using the "xengine status" command on the simulator console. Here is some example output.
Status of XEngine instance for MyRegion1 Scripts loaded : 2491 Unique scripts : 842 Scripts waiting for load : 0 Max threads : 100 Min threads : 2 Allocated threads : 36 In use threads : 18 Work items waiting : 0 Sensors : 11 Dataserver requests : 0 Timers : 36 Listeners : 59 Status of XEngine instance for MyRegion2 Scripts loaded : 0 Unique scripts : 0 Scripts waiting for load : 0 Max threads : 100 Min threads : 2 Allocated threads : 3 In use threads : 2 Work items waiting : 0 Sensors : 0 Dataserver requests : 0 Timers : 0 Listeners : 0
In this case there are two regions, MyRegion1 and MyRegion2, and hence two thread pools. The thread pool details are the lines which match the data given for the general thread pool (Unique scripts, allocated threads, etc.). In this case, the "xengine status" command also shows additional details (scripts loaded, sensors, etc.) which are not directly thread related.
