[Opensim-dev] Issues with the Simulator under high load
Justin Clark-Casey
jjustincc at googlemail.com
Sat Mar 10 02:25:20 UTC 2012
I'm sorry to say that you'll have to take the ThreadPool numbers with a very very very large pinch of salt. I believe
they only refer to the built-in mono thread pool and not the SmartThreadPool which is the one actually used (and beyond
that the core simulator and xengine use separate pools). I will try and improve this situation soon.
On 09/03/12 11:46, Akira Sonoda wrote:
> 600 on a 8 core machine ( i guess the hyperthreaded cores are visible to mono as real cores, at least in nmon they are
> reported as such ) is quite a lot of threads. This morning I went big and configured 1000, but I'm not really sure which
> approach to go...
>
> seeing the stats:
>
> Region (Close Encounter) # show threads
> 9 threads are being tracked:
> ID NAME LAST UPDATE (MS) LIFETIME (MS) PRIORITY
> STATE
> 8 PollServiceWorkerThread0 137 27996479 Lowest
> WaitSleepJoin
> 9 PollServiceWorkerThread1 27996478 27996478 Lowest
> WaitSleepJoin
> 10 PollServiceWorkerThread2 27996478 27996478 Lowest
> WaitSleepJoin
> 11 PollServiceWatcherThread 137 27996478 Lowest
> WaitSleepJoin
> 15 MapItemRequestThread (Close Encounter) 383 27992497 Lowest Background,
> WaitSleepJoin
> 18 Incoming Packets (Close Encounter) 19 27979915 Lowest
> WaitSleepJoin
> 19 Outgoing Packets (Close Encounter) 0 27979915 Lowest
> WaitSleepJoin
> 20 Heartbeat (Close Encounter) 55 27979914 Lowest
> WaitSleepJoin
> 34 AsyncLSLCmdHandlerThread 33 27964827 Lowest Background,
> WaitSleepJoin
>
> *** ThreadPool threads ***
> workers: 0 (500); ports: 0 (1000)
>
> the ThreadPool shows worker/port threads 1500 so on a 8 CPU Machine 200 MONO_THREADS_PER_CPU should be sufficient if i
> interpret those numbers correctly guessing the numbers in brackets are the max of the Pools. Therefore 1000 is possibly
> too much, Will be interesting to see if i still run into problems with 300 Threads per CPU.
>
>
>
>
> Am 9. März 2012 08:14 schrieb Dahlia Trimble <dahliatrimble at gmail.com <mailto:dahliatrimble at gmail.com>>:
>
> Sorry I don't really know much about it. In my case it was an application that used the http server dll from OpenSim
> and served probably 40-60 simultaneous requests. Mono was defaulting to 25 threads per cpu but I changed it to 75
> and I stopped having download problems. This was on a 4-core machine.
>
> I would guess if you are using 150 and seeing problems that a good place to start might be somewhere around 450-600
> and see what happens.
>
> On Thu, Mar 8, 2012 at 9:44 PM, Akira Sonoda <akira.sonoda.1 at gmail.com <mailto:akira.sonoda.1 at gmail.com>> wrote:
>
> Ooopps... my MONO_THREADS_PER_CPU=150 are obviously not enough. 2000 as stated in the article is quite a lot ...
> what are your settings? do you go with the 2000?
>
>
> Am 9. März 2012 00:07 schrieb Dahlia Trimble <dahliatrimble at gmail.com <mailto:dahliatrimble at gmail.com>>:
>
> Are you using Mono? I've seen poor performance of the http server used in OpenSimulator when insufficient
> threads are available. Manipulating the environment variable MONO_THREADS_PER_CPU has worked for me when
> I've encountered this problem before. Take a look at
> http://www.mono-project.com/Article:ThreadPool_Deadlocks for some background on this problem.
>
> As far as network performance tools go I'd probably just search the web for "network performance tool" and
> pick whatever works for you.
>
> On Thu, Mar 8, 2012 at 2:28 PM, Akira Sonoda <akira.sonoda.1 at gmail.com <mailto:akira.sonoda.1 at gmail.com>> wrote:
>
> Hi Dahlia,
>
> Am 5. März 2012 01:14 schrieb Dahlia Trimble <dahliatrimble at gmail.com <mailto:dahliatrimble at gmail.com>>:
>
> A couple thoughts, not sure if it's your problem or not.
>
> I would probably check to make sure the cache is set up properly and the file system it's on has
> plenty of space. Also make sure the disk isnt being thrashed by other processes and that the disk is
> healthy and not fragmented. There's probably some system utilities that can show disk I/O activity
> and disk health.
>
>
> There is plenty of free space on the disk.
>
> You may also have network congestion problems that could slow retrieval from the asset servers or
> slow sending of assets to other clients.
>
>
> How can i figure them out?
>
> I've made a other report from the party from Wednesday on "Pyramid at osgrid".
>
> https://docs.google.com/open?id=0B301xueh1kxdVmVZZ18tbi1TdzZ2cGlRaFhDTlo4UQ
>
> The server where "Pyramid" is located is similar to my server. The major difference is the mono version.
> There was a time when i had high quite high network load but is this network congestion?
>
> Status right now: we survive.
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
>
>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
More information about the Opensim-dev
mailing list