[Opensim-dev] Issues with the Simulator under high load
Dahlia Trimble
dahliatrimble at gmail.com
Sat Mar 10 07:56:21 UTC 2012
wouldn't the thread pool in OpenSim be using mono threads?
Also perhaps I wasn't clear, the application I found the benefit in from
increasing mono threads was *not* Opensim, but rather it was a different
application that uses the same http server as OpenSim uses.
On Fri, Mar 9, 2012 at 6:25 PM, Justin Clark-Casey <jjustincc at googlemail.com
> wrote:
> I'm sorry to say that you'll have to take the ThreadPool numbers with a
> very very very large pinch of salt. I believe they only refer to the
> built-in mono thread pool and not the SmartThreadPool which is the one
> actually used (and beyond that the core simulator and xengine use separate
> pools). I will try and improve this situation soon.
>
>
> On 09/03/12 11:46, Akira Sonoda wrote:
>
>> 600 on a 8 core machine ( i guess the hyperthreaded cores are visible to
>> mono as real cores, at least in nmon they are
>> reported as such ) is quite a lot of threads. This morning I went big and
>> configured 1000, but I'm not really sure which
>> approach to go...
>>
>> seeing the stats:
>>
>> Region (Close Encounter) # show threads
>> 9 threads are being tracked:
>> ID NAME LAST UPDATE (MS)
>> LIFETIME (MS) PRIORITY
>> STATE
>> 8 PollServiceWorkerThread0 137
>> 27996479 Lowest
>> WaitSleepJoin
>> 9 PollServiceWorkerThread1 27996478
>> 27996478 Lowest
>> WaitSleepJoin
>> 10 PollServiceWorkerThread2 27996478
>> 27996478 Lowest
>> WaitSleepJoin
>> 11 PollServiceWatcherThread 137
>> 27996478 Lowest
>> WaitSleepJoin
>> 15 MapItemRequestThread (Close Encounter) 383
>> 27992497 Lowest Background,
>> WaitSleepJoin
>> 18 Incoming Packets (Close Encounter) 19
>> 27979915 Lowest
>> WaitSleepJoin
>> 19 Outgoing Packets (Close Encounter) 0
>> 27979915 Lowest
>> WaitSleepJoin
>> 20 Heartbeat (Close Encounter) 55
>> 27979914 Lowest
>> WaitSleepJoin
>> 34 AsyncLSLCmdHandlerThread 33
>> 27964827 Lowest Background,
>> WaitSleepJoin
>>
>> *** ThreadPool threads ***
>> workers: 0 (500); ports: 0 (1000)
>>
>> the ThreadPool shows worker/port threads 1500 so on a 8 CPU Machine 200
>> MONO_THREADS_PER_CPU should be sufficient if i
>> interpret those numbers correctly guessing the numbers in brackets are
>> the max of the Pools. Therefore 1000 is possibly
>> too much, Will be interesting to see if i still run into problems with
>> 300 Threads per CPU.
>>
>>
>>
>>
>> Am 9. März 2012 08:14 schrieb Dahlia Trimble <dahliatrimble at gmail.com<mailto:
>> dahliatrimble at gmail.**com <dahliatrimble at gmail.com>>>:
>>
>>
>> Sorry I don't really know much about it. In my case it was an
>> application that used the http server dll from OpenSim
>> and served probably 40-60 simultaneous requests. Mono was defaulting
>> to 25 threads per cpu but I changed it to 75
>> and I stopped having download problems. This was on a 4-core machine.
>>
>> I would guess if you are using 150 and seeing problems that a good
>> place to start might be somewhere around 450-600
>> and see what happens.
>>
>> On Thu, Mar 8, 2012 at 9:44 PM, Akira Sonoda <akira.sonoda.1 at gmail.com<mailto:
>> akira.sonoda.1 at gmail.**com <akira.sonoda.1 at gmail.com>>> wrote:
>>
>> Ooopps... my MONO_THREADS_PER_CPU=150 are obviously not enough.
>> 2000 as stated in the article is quite a lot ...
>> what are your settings? do you go with the 2000?
>>
>>
>> Am 9. März 2012 00:07 schrieb Dahlia Trimble <
>> dahliatrimble at gmail.com <mailto:dahliatrimble at gmail.**com<dahliatrimble at gmail.com>
>> >>:
>>
>>
>> Are you using Mono? I've seen poor performance of the http
>> server used in OpenSimulator when insufficient
>> threads are available. Manipulating the environment variable
>> MONO_THREADS_PER_CPU has worked for me when
>> I've encountered this problem before. Take a look at
>> http://www.mono-project.com/**Article:ThreadPool_Deadlocks<http://www.mono-project.com/Article:ThreadPool_Deadlocks>for some background on this problem.
>>
>> As far as network performance tools go I'd probably just
>> search the web for "network performance tool" and
>> pick whatever works for you.
>>
>> On Thu, Mar 8, 2012 at 2:28 PM, Akira Sonoda <
>> akira.sonoda.1 at gmail.com <mailto:akira.sonoda.1 at gmail.**com<akira.sonoda.1 at gmail.com>>>
>> wrote:
>>
>> Hi Dahlia,
>>
>> Am 5. März 2012 01:14 schrieb Dahlia Trimble <
>> dahliatrimble at gmail.com <mailto:dahliatrimble at gmail.**com<dahliatrimble at gmail.com>
>> >>:
>>
>>
>> A couple thoughts, not sure if it's your problem or
>> not.
>>
>> I would probably check to make sure the cache is set
>> up properly and the file system it's on has
>> plenty of space. Also make sure the disk isnt being
>> thrashed by other processes and that the disk is
>> healthy and not fragmented. There's probably some
>> system utilities that can show disk I/O activity
>> and disk health.
>>
>>
>> There is plenty of free space on the disk.
>>
>> You may also have network congestion problems that
>> could slow retrieval from the asset servers or
>> slow sending of assets to other clients.
>>
>>
>> How can i figure them out?
>>
>> I've made a other report from the party from Wednesday on
>> "Pyramid at osgrid".
>>
>> https://docs.google.com/open?**id=**
>> 0B301xueh1kxdVmVZZ18tbi1TdzZ2c**GlRaFhDTlo4UQ<https://docs.google.com/open?id=0B301xueh1kxdVmVZZ18tbi1TdzZ2cGlRaFhDTlo4UQ>
>>
>> The server where "Pyramid" is located is similar to my
>> server. The major difference is the mono version.
>> There was a time when i had high quite high network load
>> but is this network congestion?
>>
>> Status right now: we survive.
>>
>>
>>
>> ______________________________**_________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.**
>> berlios.de <Opensim-dev at lists.berlios.de>>
>>
>> https://lists.berlios.de/**mailman/listinfo/opensim-dev<https://lists.berlios.de/mailman/listinfo/opensim-dev>
>>
>>
>>
>> ______________________________**_________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.**
>> berlios.de <Opensim-dev at lists.berlios.de>>
>>
>> https://lists.berlios.de/**mailman/listinfo/opensim-dev<https://lists.berlios.de/mailman/listinfo/opensim-dev>
>>
>>
>>
>> ______________________________**_________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.**
>> berlios.de <Opensim-dev at lists.berlios.de>>
>>
>> https://lists.berlios.de/**mailman/listinfo/opensim-dev<https://lists.berlios.de/mailman/listinfo/opensim-dev>
>>
>>
>>
>> ______________________________**_________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.**berlios.de<Opensim-dev at lists.berlios.de>
>> >
>>
>> https://lists.berlios.de/**mailman/listinfo/opensim-dev<https://lists.berlios.de/mailman/listinfo/opensim-dev>
>>
>>
>>
>>
>> ______________________________**_________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/**mailman/listinfo/opensim-dev<https://lists.berlios.de/mailman/listinfo/opensim-dev>
>>
>
>
> --
> Justin Clark-Casey (justincc)
> http://justincc.org/blog
> http://twitter.com/justincc
>
> ______________________________**_________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/**mailman/listinfo/opensim-dev<https://lists.berlios.de/mailman/listinfo/opensim-dev>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-dev/attachments/20120309/8ef5c005/attachment-0001.html>
More information about the Opensim-dev
mailing list