[Opensim-dev] Question about Threads Use in OpenSim ...
Ursula MATOVA
ursula.matova at klintcentral.net
Wed Apr 9 17:52:33 UTC 2008
[[ Part #1 ]]
INFO: I had a configuration error in my OpenSim.ini ... I used the
same "http_listener_port" for all servers ( Port 9000 ).
I changed that ( see bellow ) and seems that solved the CPU Usage
problem on my 1st Region Server.
Sorry for EMail pollution :)
Here are my notes ( with the wrong configuration ) :
Regards,
Ursula.
Initial Server Boot :
xentop - 18:50:37 Xen 3.1.0
7 domains: 1 running, 6 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 136 1.0 393216 10.9 no
limit n/a 2 4 0 2 0 0
0 0 0
osg200 --b--- 1 0.0 524288 14.5 524288
14.5 1 1 14 19 2 0 52 725 0
osg201 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 1 2 0 0 13 0
osg203 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 1 2 0 0 13 0
osg204 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 1 2 0 0 18 0
osg205 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 1 2 0 0 13 0
osg206 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 1 2 0 0 13 0
OSG201: OpenSim U+G+A+I Started :
xentop - 18:53:38 Xen 3.1.0
7 domains: 1 running, 6 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 138 1.2 393216 10.9 no
limit n/a 2 4 0 2 0 0
0 0 0
osg200 --b--- 1 0.1 524288 14.5 524288
14.5 1 1 158 6060 2 0 302 1413 0
osg201 --b--- 4 0.1 524288 14.5 524288
14.5 1 1 6230 237 2 0 829 351 0
osg203 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 2 2 0 0 13 0
osg204 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 2 2 0 0 18 0
osg205 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 2 2 0 0 13 0
osg206 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 2 2 0 0 13 0
top - 18:15:04 up 42 min, 5 users, load average: 0.02, 0.01, 0.00
Tasks: 54 total, 2 running, 52 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi,
0.0%si, 0.0%st
Mem: 524436k total, 102520k used, 421916k free, 3352k buffers
Swap: 262136k total, 0k used, 262136k free, 33872k cached
OSG203 : Start First "Region Server" (S) :
xentop - 18:55:09 Xen 3.1.0
7 domains: 1 running, 6 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 139 1.5 393216 10.9 no
limit n/a 2 4 0 2 0 0
0 0 0
osg200 --b--- 1 0.1 524288 14.5 524288
14.5 1 1 4109 12138 2 0 512 2236 0
osg201 --b--- 4 0.1 524288 14.5 524288
14.5 1 1 6402 413 2 0 845 437 0
osg203 --b--- 5 0.6 524288 14.5 524288
14.5 1 1 6450 4167 2 0 1555 216 0
osg204 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 2 2 0 0 18 0
osg205 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 2 2 0 0 13 0
osg206 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 2 2 0 0 13 0
top - 18:16:35 up 1:17, 2 users, load average: 0.04, 0.01, 0.00
Tasks: 42 total, 2 running, 40 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi,
0.0%si, 0.0%st
Mem: 524436k total, 122876k used, 401560k free, 3724k buffers
Swap: 262136k total, 0k used, 262136k free, 50196k cached
OSG204 : Start 2nd "Region Server" (S) :
entop - 18:56:37 Xen 3.1.0
7 domains: 1 running, 6 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 141 1.2 393216 10.9 no
limit n/a 2 4 0 3 0 0
0 0 0
osg200 --b--- 2 0.1 524288 14.5 524288
14.5 1 1 8112 18225 2 0 529 2998 0
osg201 --b--- 5 0.1 524288 14.5 524288
14.5 1 1 6694 634 2 0 847 476 0
osg203 --b--- 5 0.5 524288 14.5 524288
14.5 1 1 6477 4182 2 0 1562 307 0
osg204 --b--- 5 0.7 524288 14.5 524288
14.5 1 1 6483 4325 2 0 1550 223 0
osg205 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 2 2 0 20 44 0
osg206 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 2 2 0 20 44 0
top - 18:18:06 up 1:19, 2 users, load average: 0.04, 0.01, 0.00
Tasks: 42 total, 2 running, 40 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi,
0.0%si, 0.0%st
Mem: 524436k total, 123744k used, 400692k free, 3740k buffers
Swap: 262136k total, 0k used, 262136k free, 50276k cached
OSG205 : Start 3rd "Region Server" (S) :
xentop - 19:06:29 Xen 3.1.0
7 domains: 1 running, 6 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 148 1.2 393216 10.9 no
limit n/a 2 4 0 3 0 0
0 0 0
osg200 --b--- 2 0.1 524288 14.5 524288
14.5 1 1 12110 24314 2 0 580 3950 0
osg201 --b--- 6 0.2 524288 14.5 524288
14.5 1 1 6989 859 2 0 847 480 0
osg203 --b--- 113 0.6 524288 14.5 524288
14.5 1 1 6620 4796 2 0 3233 2544 0
osg204 --b--- 11 0.6 524288 14.5 524288
14.5 1 1 6547 4421 2 0 3006 1155 0
osg205 --b--- 5 0.6 524288 14.5 524288
14.5 1 1 6449 4318 2 0 1546 104 0
osg206 --b--- 0 0.0 524288 14.5 524288
14.5 1 1 0 3 2 0 20 49 0
top - 18:28:11 up 1:29, 2 users, load average: 0.04, 0.01, 0.00
Tasks: 42 total, 2 running, 40 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi,
0.0%si, 0.3%st
Mem: 524436k total, 119784k used, 404652k free, 3760k buffers
Swap: 262136k total, 0k used, 262136k free, 50208k cached
OSG206 : Start 4th "Region Server" (S) :
xentop - 19:07:57 Xen 3.1.0
7 domains: 1 running, 6 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 150 1.2 393216 10.9 no
limit n/a 2 4 0 4 0 0
0 0 0
osg200 --b--- 3 0.1 524288 14.5 524288
14.5 1 1 16110 30396 2 0 595 4747 0
osg201 --b--- 6 0.1 524288 14.5 524288
14.5 1 1 7284 1082 2 0 847 487 0
osg203 --b--- 114 0.7 524288 14.5 524288
14.5 1 1 6620 4796 2 0 3233 2553 0
osg204 --b--- 12 0.7 524288 14.5 524288
14.5 1 1 6548 4422 2 0 3006 1160 0
osg205 --b--- 6 0.8 524288 14.5 524288
14.5 1 1 6471 4332 2 0 1551 328 0
osg206 --b--- 5 0.6 524288 14.5 524288
14.5 1 1 6480 4326 2 0 1544 108 0
top - 18:29:28 up 1:30, 2 users, load average: 0.21, 0.07, 0.02
Tasks: 42 total, 2 running, 40 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi,
0.0%si, 0.0%st
Mem: 524436k total, 124356k used, 400080k free, 3756k buffers
Swap: 262136k total, 0k used, 262136k free, 50256k cached
Le 9 avr. 08 à 19:23, Ursula MATOVA a écrit :
> Ouch,
>
> Please, forgot my idea about port 9000 ... :S
>
> It's normal that I only use port 9000 on my first Region Server : I
> have a NAT (RDR) rule for that :) (( Sometimes, i'm really stupid ! ))
>
> But, I'll post you some metrics, could help you :)
>
> Regards,
>
> Ursula.
>
> Le 9 avr. 08 à 18:19, Ursula MATOVA a écrit :
>
>> Hi Justin,
>>
>> Yes, it was a freeze with 100% CPU ( on one core ) ... But, not using
>> mono 1.9 at this time. When I upgrade to Mono 1.9, was much better :)
>>
>> I'll continue my tests with SVN.4111 before trying the latest SVN
>> release.
>>
>> I'll let you know in a while.
>>
>> Regards,
>>
>> Ursula.
>>
>> Le 9 avr. 08 à 18:14, Justin Clark-Casey a écrit :
>>
>>> Is this a freeze with 100% cpu? sdague has been having some
>>> mono-1.9
>>> issues in this area.
>>>
>>> Alternatively, I've also seen Wright Plaza on osgrid deadlock once
>>> (with
>>> idling cpu levels), which might be something to do with people
>>> logging
>>> in to a specific location selected in the login screen (I'm a bit
>>> hazy
>>> on the details since I haven't chased this one up).
>>>
>>> --
>>> justincc
>>>
>>>
>>> Ursula MATOVA wrote:
>>>> Hi Stefan,
>>>>
>>>> Well, I'm now testing SVN.4111 and it seems to work better ...
>>>> Sometimes the Sim freeze, but most often in Standalone mode.
>>>>
>>>> For now, I can run OpenSim, with acceptable response time in 4x4
>>>> Regions.
>>>>
>>>> And I had an Idea the past night :)
>>>>
>>>> I think the component that use too many CPU is the one that deal
>>>> with
>>>> port 9000 ( mean something in OpenSim.exe ).
>>>>
>>>> Here is what I'm thinking that :
>>>>
>>>> As I mention in Mantis #909, I'm running OpenSim in Grid mode, 4x4
>>>> Regions, BUT, I have created 6 Virtual Servers with Xen.
>>>>
>>>> Linux Debian 4.0r3 Etch, Mono 1.9 / Xen 3.1.0.
>>>>
>>>> Here is the Topology :
>>>>
>>>> Dom0 ( 256Mo ) <-- Main Host ( Nothing on it except xen ),
>>>> DomU ( osg200 / 256MB Ram ) <-- Apache + MySQL ( Maybe Squid in the
>>>> future ),
>>>> DomU ( osg201 / 256MB Ram ) <-- OpenSim "U+G+A+I", ( Will update it
>>>> to
>>>> 512MB Ram )
>>>> DomU ( osg203 / 512MB Ram ) <-- OpenSim "S" ( Region Server, 4
>>>> Regions ),
>>>> DomU ( osg204 / 512MB Ram ) <-- OpenSim "S" ( Region Server, 4
>>>> Regions ),
>>>> DomU ( osg205 / 512MB Ram ) <-- OpenSim "S" ( Region Server, 4
>>>> Regions ),
>>>> DomU ( osg206 / 512MB Ram ) <-- OpenSim "S" ( Region Server, 4
>>>> Regions ),
>>>>
>>>> The startup order is : osg200 > osg201 ( UGAI ) > osg203 (S) >
>>>> osg204 ...
>>>>
>>>> While monitoring my firewall logs, I found that even if my avatar
>>>> is
>>>> located in a region that belongs to OSG204( for example ) ... There
>>>> was Network Trafic to port 9000 ( OSG203 -- The first Region
>>>> server I
>>>> started ).
>>>> And, while monitoring RAM/CPU Usage on my Virtual Servers, I have
>>>> noticed that OSG203 is in "High CPU usage" state ( much more that
>>>> OSG204, 205 or 206 )
>>>>
>>>> Of course I have to confirm all those points and drop you some
>>>> metrics. And I have some more tests to prove that. But it could
>>>> be a
>>>> good start point for investigation.
>>>>
>>>> I'll make tests tonight and post the results there.
>>>>
>>>> Regards,
>>>>
>>>> Ursula.
>>>>
>>>> (( Ouch, my english turns bad :D Sorry. ))
>>>>
>>>>
>>>> Le 9 avr. 08 à 17:48, Stefan Andersson a écrit :
>>>>> What's the current status on the thread issues? Are we looking
>>>>> into
>>>>> it? Have we identified consumers/culprits?
>>>>>
>>>>> Best,
>>>>> /Stefan
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>>> From: ursula.matova at klintcentral.net
>>>>> <mailto:ursula.matova at klintcentral.net>
>>>>>> To: opensim-dev at lists.berlios.de <mailto:opensim-dev at lists.berlios.de
>>>>>>>
>>>>>> Date: Sun, 17 Feb 2008 11:09:27 +0100
>>>>>> Subject: [Opensim-dev] Question about Threads Use in OpenSim ...
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> First, please be indulgent about my poor english :) You all know
>>>>>> how
>>>>>> french people are bad with foreign languages ...
>>>>>> Thanks in advance.
>>>>>>
>>>>>> Well, I'm running OpenSim in StandAlone mode, 3x3 Regions ( as
>>>>>> Grid
>>>>>> mode is so difficult to configure behind a firewall ) ...
>>>>>> Using a home made "Linux From Scratch" box with Mono 1.2.6
>>>>>> ( daily
>>>>>> snapshot ) ...
>>>>>> The computer is an Athlon64X2 4400+ with 4GB Ram ... ( is it
>>>>>> large
>>>>>> enough ? )
>>>>>>
>>>>>> In the past ( before SVN.3335 or something ) when running
>>>>>> OpenSim,
>>>>>> just after the simulator goes Up, the OpenSim.exe process use
>>>>>> around
>>>>>> 96 threads and everything works fine,
>>>>>> But, now, I saw that this number of threads reach 118 :(
>>>>>> As I posted in the past in Mantis that Mono under Linux is really
>>>>>> bad
>>>>>> ( performances & coredump ) when you reach the limit of 170
>>>>>> threads ...
>>>>>>
>>>>>> When I connect 1 user to the sim : Thread count is now reaching :
>>>>>> 157
>>>>>> When I connect a 2nd user to the sim : Thread count reach : 200
>>>>>> ( in that case : 200 threads used, the sim doesn't crash like in
>>>>>> the
>>>>>> past but I find everything slow ... and my CPU is now 100% use /
>>>>>> 7%
>>>>>> Ram used )
>>>>>>
>>>>>> Then, that mean, each time a user is connecting to the Sim, it
>>>>>> use
>>>>>> about 40 threads ...
>>>>>> So, if I like to connect 20 users, I need around 40x20 threads =
>>>>>> 800
>>>>>> threads ...
>>>>>> And I'm not sure at all that Mono could manage that :(
>>>>>>
>>>>>> What's you're feeling on that point ?
>>>>>>
>>>>>> So, my question is : why not forking a new "OpenSim.exe" process
>>>>>> for
>>>>>> each new connected user ?
>>>>>> Of course it's mostly for the standalone mode ( but the problem
>>>>>> is a
>>>>>> bit the same in Grid Mode ) ...
>>>>>>
>>>>>> Maybe it's a simple/stupid idea from me, because I haven't enough
>>>>>> development skills to help all the people working hard to bring
>>>>>> up
>>>>>> this marvelous simulator.
>>>>>>
>>>>>> Please, just let me know your position about it.
>>>>>>
>>>>>> Thanks a lot guys for all your dev,
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Ursula Matova
>>>>>> ( Of course this is my S/L Avatar Name :) )
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Opensim-dev mailing list
>>>>>> Opensim-dev at lists.berlios.de <mailto:Opensim-
>>>>>> dev at lists.berlios.de>
>>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>>
>>>>> _______________________________________________
>>>>> Opensim-dev mailing list
>>>>> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de>
>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Opensim-dev mailing list
>>>> Opensim-dev at lists.berlios.de
>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>
>>>
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
More information about the Opensim-dev
mailing list