[Opensim-dev] Question about Threads Use in OpenSim ...
Ursula MATOVA
ursula.matova at klintcentral.net
Wed Apr 9 17:54:32 UTC 2008
[[Part #2]]
Connecting to the Grid ( port 8002 ) with S/L Client 1.19.1.4 :
After 2 or 3 minutes, once logged to a Region that belongs to OSG203
( My First Region Server ) :
xentop - 19:13:24 Xen 3.1.0
7 domains: 1 running, 6 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 153 0.8 393216 10.9 no
limit n/a 2 4 0 4 0 0
0 0 0
osg200 --b--- 3 0.1 524288 14.5 524288
14.5 1 1 18677 30460 2 0 671 4970 0
osg201 --b--- 7 0.1 524288 14.5 524288
14.5 1 1 7499 1170 2 0 892 525 0
osg203 --b--- 117 1.2 524288 14.5 524288
14.5 1 1 7096 5328 2 0 3248 2600 0
osg204 --b--- 15 0.9 524288 14.5 524288
14.5 1 1 6612 4462 2 0 3008 1171 0
osg205 --b--- 8 0.8 524288 14.5 524288
14.5 1 1 6472 4333 2 0 1551 330 0
osg206 --b--- 8 0.8 524288 14.5 524288
14.5 1 1 6497 4333 2 0 1547 305 0
Move to a Region that belongs to OSG204 ( 2nd Region Server ),
-- During MiniMap Display :
xentop - 19:19:01 Xen 3.1.0
7 domains: 2 running, 4 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 157 1.5 393216 10.9 no
limit n/a 2 4 0 5 0 0
0 0 0
osg200 --b--- 3 0.1 524288 14.5 524288
14.5 1 1 18995 30505 2 0 695 5190 0
osg201 ------ 8 0.2 524288 14.5 524288
14.5 1 1 8010 1489 2 0 892 549 0
osg203 --b--- 120 0.8 524288 14.5 524288
14.5 1 1 7222 5522 2 0 3248 2625 0
osg204 --b--- 19 0.9 524288 14.5 524288
14.5 1 1 6863 4843 2 0 3009 1204 0
osg205 -----r 97 99.3 524288 14.5 524288
14.5 1 1 6696 4704 2 0 1555 365 0
osg206 --b--- 11 0.9 524288 14.5 524288
14.5 1 1 6619 4399 2 0 1549 323 0
xentop - 19:17:45 Xen 3.1.0
7 domains: 1 running, 6 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 156 1.3 393216 10.9 no
limit n/a 2 4 0 4 0 0
0 0 0
osg200 --b--- 3 0.1 524288 14.5 524288
14.5 1 1 18791 30477 2 0 690 5098 0
osg201 --b--- 7 0.1 524288 14.5 524288
14.5 1 1 7665 1281 2 0 892 533 0
osg203 --b--- 120 0.8 524288 14.5 524288
14.5 1 1 7185 5485 2 0 3248 2612 0
osg204 --b--- 18 1.3 524288 14.5 524288
14.5 1 1 6750 4698 2 0 3009 1182 0
osg205 --b--- 70 0.9 524288 14.5 524288
14.5 1 1 6566 4396 2 0 1553 342 0
osg206 --b--- 9 0.8 524288 14.5 524288
14.5 1 1 6497 4333 2 0 1547 305 0
.../... Flying around my Regions ...
Then on OSG203 :
xentop - 19:26:40 Xen 3.1.0
7 domains: 1 running, 6 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 162 1.1 393216 10.9 no
limit n/a 2 4 0 5 0 0
0 0 0
osg200 --b--- 4 0.1 524288 14.5 524288
14.5 1 1 18997 30508 2 0 728 5424 0
osg201 --b--- 9 0.2 524288 14.5 524288
14.5 1 1 8010 1489 2 0 892 549 0
osg203 --b--- 250 42.2 524288 14.5 524288
14.5 1 1 7250 5577 2 0 3248 2690 0
osg204 --b--- 24 1.2 524288 14.5 524288
14.5 1 1 6890 4898 2 0 3009 1263 0
osg205 --b--- 110 1.4 524288 14.5 524288
14.5 1 1 6753 5004 2 0 1555 365 0
osg206 --b--- 15 1.0 524288 14.5 524288
14.5 1 1 6644 4447 2 0 1549 324 0
OHHHHHHHHHHH !
I got it ... I had an error in my OpenSim.ini :
1st Region Server ( Regions ports are : 9000, 9001, 9002 and 9003 ),
so, I set "HTTP_LISTENER_PORT" to 9000,
2nd Region Server ( Regions ports are : 9004 -> 9007), so, I set
"HTTP_PORT" to 9004,
3rd Region Server ( Regions ports are : 9008 -> 9011), so, I set
"HTTP_PORT" to 9008,
4th Region Server ( Regions ports are : 9012 -> 9015), so, I set
"HTTP_PORT" to 9012,
Now everything seems to be ok :
xentop - 19:39:09 Xen 3.1.0
7 domains: 2 running, 5 blocked, 0 paused, 0 crashed, 0 dying, 0
shutdown
Mem: 3604028k total, 3576064k used, 27964k free CPUs: 2 @ 2210MHz
NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k)
MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS VBD_OO VBD_RD VBD_WR
SSID
Domain-0 -----r 171 1.3 393216 10.9 no
limit n/a 2 4 0 6 0 0
0 0 0
osg200 -----r 5 0.1 524288 14.5 524288
14.5 1 1 36790 54951 2 0 823 8988 0
osg201 --b--- 11 0.2 524288 14.5 524288
14.5 1 1 10153 2894 2 0 892 641 0
osg203 --b--- 328 0.8 524288 14.5 524288
14.5 1 1 14349 10611 2 0 3249 3049 0
osg204 --b--- 39 0.9 524288 14.5 524288
14.5 1 1 13919 9670 2 0 3047 1691 0
osg205 --b--- 124 1.5 524288 14.5 524288
14.5 1 1 13524 9925 2 0 1593 678 0
osg206 --b--- 35 2.0 524288 14.5 524288
14.5 1 1 13241 8876 2 0 1579 684 0
CPU(s) never show more than 10% ( and mostly are around 5% )
Now : ABOUT THREADS
OSG201 (UGAI) :
root 1850 1846 1850 0 8 18:13 pts/0 00:00:00 mono
OpenSim.Grid.UserServer.exe --> 7 Threads
root 1870 1866 1870 0 8 18:13 pts/1 00:00:00 mono
OpenSim.Grid.GridServer.exe --> 7 Threads,
root 1891 1887 1891 0 8 18:13 pts/2 00:00:01 mono
OpenSim.Grid.AssetServer.exe --> 7 Threads,
root 1911 1907 1911 0 7 18:14 pts/3 00:00:00 mono
OpenSim.Grid.InventoryServer.exe --> 6 Threads
OSG203 (S) :
mono OpenSim.exe --> 54 Threads
OSG204 (S) :
mono OpenSim.exe --> 53 Threads
OSG205 (S) :
mono OpenSim.exe --> 53 Threads
OSG206 (S) :
mono OpenSim.exe --> 53 Threads
Now, I'll let the whole servers run, and wait to see if the CPU Usage
is stable
Regards,
Ursula ...
Le 9 avr. 08 à 19:23, Ursula MATOVA a écrit :
> Ouch,
>
> Please, forgot my idea about port 9000 ... :S
>
> It's normal that I only use port 9000 on my first Region Server : I
> have a NAT (RDR) rule for that :) (( Sometimes, i'm really stupid ! ))
>
> But, I'll post you some metrics, could help you :)
>
> Regards,
>
> Ursula.
>
> Le 9 avr. 08 à 18:19, Ursula MATOVA a écrit :
>
>> Hi Justin,
>>
>> Yes, it was a freeze with 100% CPU ( on one core ) ... But, not using
>> mono 1.9 at this time. When I upgrade to Mono 1.9, was much better :)
>>
>> I'll continue my tests with SVN.4111 before trying the latest SVN
>> release.
>>
>> I'll let you know in a while.
>>
>> Regards,
>>
>> Ursula.
>>
>> Le 9 avr. 08 à 18:14, Justin Clark-Casey a écrit :
>>
>>> Is this a freeze with 100% cpu? sdague has been having some
>>> mono-1.9
>>> issues in this area.
>>>
>>> Alternatively, I've also seen Wright Plaza on osgrid deadlock once
>>> (with
>>> idling cpu levels), which might be something to do with people
>>> logging
>>> in to a specific location selected in the login screen (I'm a bit
>>> hazy
>>> on the details since I haven't chased this one up).
>>>
>>> --
>>> justincc
>>>
>>>
>>> Ursula MATOVA wrote:
>>>> Hi Stefan,
>>>>
>>>> Well, I'm now testing SVN.4111 and it seems to work better ...
>>>> Sometimes the Sim freeze, but most often in Standalone mode.
>>>>
>>>> For now, I can run OpenSim, with acceptable response time in 4x4
>>>> Regions.
>>>>
>>>> And I had an Idea the past night :)
>>>>
>>>> I think the component that use too many CPU is the one that deal
>>>> with
>>>> port 9000 ( mean something in OpenSim.exe ).
>>>>
>>>> Here is what I'm thinking that :
>>>>
>>>> As I mention in Mantis #909, I'm running OpenSim in Grid mode, 4x4
>>>> Regions, BUT, I have created 6 Virtual Servers with Xen.
>>>>
>>>> Linux Debian 4.0r3 Etch, Mono 1.9 / Xen 3.1.0.
>>>>
>>>> Here is the Topology :
>>>>
>>>> Dom0 ( 256Mo ) <-- Main Host ( Nothing on it except xen ),
>>>> DomU ( osg200 / 256MB Ram ) <-- Apache + MySQL ( Maybe Squid in the
>>>> future ),
>>>> DomU ( osg201 / 256MB Ram ) <-- OpenSim "U+G+A+I", ( Will update it
>>>> to
>>>> 512MB Ram )
>>>> DomU ( osg203 / 512MB Ram ) <-- OpenSim "S" ( Region Server, 4
>>>> Regions ),
>>>> DomU ( osg204 / 512MB Ram ) <-- OpenSim "S" ( Region Server, 4
>>>> Regions ),
>>>> DomU ( osg205 / 512MB Ram ) <-- OpenSim "S" ( Region Server, 4
>>>> Regions ),
>>>> DomU ( osg206 / 512MB Ram ) <-- OpenSim "S" ( Region Server, 4
>>>> Regions ),
>>>>
>>>> The startup order is : osg200 > osg201 ( UGAI ) > osg203 (S) >
>>>> osg204 ...
>>>>
>>>> While monitoring my firewall logs, I found that even if my avatar
>>>> is
>>>> located in a region that belongs to OSG204( for example ) ... There
>>>> was Network Trafic to port 9000 ( OSG203 -- The first Region
>>>> server I
>>>> started ).
>>>> And, while monitoring RAM/CPU Usage on my Virtual Servers, I have
>>>> noticed that OSG203 is in "High CPU usage" state ( much more that
>>>> OSG204, 205 or 206 )
>>>>
>>>> Of course I have to confirm all those points and drop you some
>>>> metrics. And I have some more tests to prove that. But it could
>>>> be a
>>>> good start point for investigation.
>>>>
>>>> I'll make tests tonight and post the results there.
>>>>
>>>> Regards,
>>>>
>>>> Ursula.
>>>>
>>>> (( Ouch, my english turns bad :D Sorry. ))
>>>>
>>>>
>>>> Le 9 avr. 08 à 17:48, Stefan Andersson a écrit :
>>>>> What's the current status on the thread issues? Are we looking
>>>>> into
>>>>> it? Have we identified consumers/culprits?
>>>>>
>>>>> Best,
>>>>> /Stefan
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------
>>>>>
>>>>>> From: ursula.matova at klintcentral.net
>>>>> <mailto:ursula.matova at klintcentral.net>
>>>>>> To: opensim-dev at lists.berlios.de <mailto:opensim-dev at lists.berlios.de
>>>>>>>
>>>>>> Date: Sun, 17 Feb 2008 11:09:27 +0100
>>>>>> Subject: [Opensim-dev] Question about Threads Use in OpenSim ...
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> First, please be indulgent about my poor english :) You all know
>>>>>> how
>>>>>> french people are bad with foreign languages ...
>>>>>> Thanks in advance.
>>>>>>
>>>>>> Well, I'm running OpenSim in StandAlone mode, 3x3 Regions ( as
>>>>>> Grid
>>>>>> mode is so difficult to configure behind a firewall ) ...
>>>>>> Using a home made "Linux From Scratch" box with Mono 1.2.6
>>>>>> ( daily
>>>>>> snapshot ) ...
>>>>>> The computer is an Athlon64X2 4400+ with 4GB Ram ... ( is it
>>>>>> large
>>>>>> enough ? )
>>>>>>
>>>>>> In the past ( before SVN.3335 or something ) when running
>>>>>> OpenSim,
>>>>>> just after the simulator goes Up, the OpenSim.exe process use
>>>>>> around
>>>>>> 96 threads and everything works fine,
>>>>>> But, now, I saw that this number of threads reach 118 :(
>>>>>> As I posted in the past in Mantis that Mono under Linux is really
>>>>>> bad
>>>>>> ( performances & coredump ) when you reach the limit of 170
>>>>>> threads ...
>>>>>>
>>>>>> When I connect 1 user to the sim : Thread count is now reaching :
>>>>>> 157
>>>>>> When I connect a 2nd user to the sim : Thread count reach : 200
>>>>>> ( in that case : 200 threads used, the sim doesn't crash like in
>>>>>> the
>>>>>> past but I find everything slow ... and my CPU is now 100% use /
>>>>>> 7%
>>>>>> Ram used )
>>>>>>
>>>>>> Then, that mean, each time a user is connecting to the Sim, it
>>>>>> use
>>>>>> about 40 threads ...
>>>>>> So, if I like to connect 20 users, I need around 40x20 threads =
>>>>>> 800
>>>>>> threads ...
>>>>>> And I'm not sure at all that Mono could manage that :(
>>>>>>
>>>>>> What's you're feeling on that point ?
>>>>>>
>>>>>> So, my question is : why not forking a new "OpenSim.exe" process
>>>>>> for
>>>>>> each new connected user ?
>>>>>> Of course it's mostly for the standalone mode ( but the problem
>>>>>> is a
>>>>>> bit the same in Grid Mode ) ...
>>>>>>
>>>>>> Maybe it's a simple/stupid idea from me, because I haven't enough
>>>>>> development skills to help all the people working hard to bring
>>>>>> up
>>>>>> this marvelous simulator.
>>>>>>
>>>>>> Please, just let me know your position about it.
>>>>>>
>>>>>> Thanks a lot guys for all your dev,
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Ursula Matova
>>>>>> ( Of course this is my S/L Avatar Name :) )
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Opensim-dev mailing list
>>>>>> Opensim-dev at lists.berlios.de <mailto:Opensim-
>>>>>> dev at lists.berlios.de>
>>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>>
>>>>> _______________________________________________
>>>>> Opensim-dev mailing list
>>>>> Opensim-dev at lists.berlios.de <mailto:Opensim-dev at lists.berlios.de>
>>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Opensim-dev mailing list
>>>> Opensim-dev at lists.berlios.de
>>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>>
>>>
>>> _______________________________________________
>>> Opensim-dev mailing list
>>> Opensim-dev at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>>
>>
>> _______________________________________________
>> Opensim-dev mailing list
>> Opensim-dev at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-dev
>>
>
> _______________________________________________
> Opensim-dev mailing list
> Opensim-dev at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-dev
>
More information about the Opensim-dev
mailing list