[Opensim-dev] Issues with the Simulator under high load

Fri Mar 9 11:46:17 UTC 2012

600 on a 8 core machine ( i guess the hyperthreaded cores are visible to
mono as real cores, at least in nmon they are reported as such ) is quite a
lot of threads. This morning I went big and configured 1000, but I'm not
really sure which approach to go...

seeing the stats:

Region (Close Encounter) # show threads
9 threads are being tracked:
    ID                                  NAME   LAST UPDATE (MS)   LIFETIME
(MS)     PRIORITY                            STATE
     8              PollServiceWorkerThread0                137
27996479       Lowest                    WaitSleepJoin
     9              PollServiceWorkerThread1           27996478
27996478       Lowest                    WaitSleepJoin
    10              PollServiceWorkerThread2           27996478
27996478       Lowest                    WaitSleepJoin
    11              PollServiceWatcherThread                137
27996478       Lowest                    WaitSleepJoin
    15   MapItemRequestThread (Close Encounter)                383
27992497       Lowest        Background, WaitSleepJoin
    18    Incoming Packets (Close Encounter)                 19
27979915       Lowest                    WaitSleepJoin
    19    Outgoing Packets (Close Encounter)                  0
27979915       Lowest                    WaitSleepJoin
    20           Heartbeat (Close Encounter)                 55
27979914       Lowest                    WaitSleepJoin
    34              AsyncLSLCmdHandlerThread                 33
27964827       Lowest        Background, WaitSleepJoin

*** ThreadPool threads ***
workers: 0 (500); ports: 0 (1000)

the ThreadPool shows worker/port threads 1500 so on a 8 CPU Machine 200
MONO_THREADS_PER_CPU should be sufficient if i interpret those numbers
correctly guessing the numbers in brackets are the max of the Pools.
Therefore 1000 is possibly too much, Will be interesting to  see if i still
run into problems with 300 Threads per CPU.

Am 9. März 2012 08:14 schrieb Dahlia Trimble <dahliatrimble at gmail.com>:

> Sorry I don't really know much about it. In my case it was an application
> that used the http server dll from OpenSim and served probably 40-60
> simultaneous requests. Mono was defaulting to 25 threads per cpu but I
> changed it to 75 and I stopped having download problems. This was on a
> 4-core machine.
>
> I would guess if you are using 150 and seeing problems that a good place
> to start might be somewhere around 450-600 and see what happens.
>
> On Thu, Mar 8, 2012 at 9:44 PM, Akira Sonoda <akira.sonoda.1 at gmail.com>wrote:
>
>> Ooopps... my MONO_THREADS_PER_CPU=150 are obviously not enough. 2000 as
>> stated in the article is quite a lot ... what are your settings? do you go
>> with the 2000?
>>
>>
>> Am 9. März 2012 00:07 schrieb Dahlia Trimble <dahliatrimble at gmail.com>:
>>
>> Are you using Mono? I've seen poor performance of the http server used in
>>> OpenSimulator when insufficient threads are available. Manipulating the
>>> environment variable MONO_THREADS_PER_CPU has worked for me when I've
>>> encountered this problem before. Take a look at
>>> http://www.mono-project.com/Article:ThreadPool_Deadlocks for some
>>> background on this problem.
>>>
>>> As far as network performance tools go I'd probably just search the web
>>> for "network performance tool" and pick whatever works for you.
>>>
>>> On Thu, Mar 8, 2012 at 2:28 PM, Akira Sonoda <akira.sonoda.1 at gmail.com>wrote:
>>>
>>>> Hi Dahlia,
>>>>
>>>> Am 5. März 2012 01:14 schrieb Dahlia Trimble <dahliatrimble at gmail.com>:
>>>>
>>>> A couple thoughts, not sure if it's your problem or not.
>>>>>
>>>>> I would probably check to make sure the cache is set up properly and
>>>>> the file system it's on has plenty of space. Also make sure the disk isnt
>>>>> being thrashed by other processes and that the disk is healthy and not
>>>>> fragmented. There's probably some system utilities that can show disk I/O
>>>>> activity and disk health.
>>>>>
>>>>
>>>> There is plenty of free space on the disk.
>>>>
>>>>
>>>>> You may also have network congestion problems that could slow
>>>>> retrieval from the asset servers or slow sending of assets to other clients.
>>>>>
>>>>
>>>> How can i figure them out?
>>>>
>>>> I've made a other report from the party from Wednesday on
>>>> "Pyramid at osgrid".
>>>>
>>>>
>>>> https://docs.google.com/open?id=0B301xueh1kxdVmVZZ18tbi1TdzZ2cGlRaFhDTlo4UQ
>>>>
>>>> The server where "Pyramid" is located is similar to my server. The
>>>> major difference is the mono version. There was a time when i had high
>>>> quite high network load but is this network congestion?
>>>>
>>>> Status right now: we survive.
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Opensim-dev mailing list
>>>> Opensim-dev at lists.berlios.de
>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>
>>>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20120309/fff4be3d/attachment-0001.html>