0003353: 100% CPU usage from the mono process

(0010178)
Jamenai (reporter)
2009-03-28 05:58

Build 8916

(0010179)
Teravus (administrator)
2009-03-28 06:01

I hate to say it, .. and I will. Unless you can provide a better bug report, chances are, this one will be ignored. It isn't a personal attack or anythin, it's just that with the information provided in this report, there's really nothing that we can do. We don't know enough about your system or the situation in which it occurs to be able to do anything about it.

(0010180)
Jamenai (reporter)
2009-03-28 06:08
edited on: 2009-03-28 06:10

hi, if you need any information i can look what i can do.
The last versions before 8862 are stable.

If any other people have the same problem they can tell it and we get more informations.

But its a bug or not?

And if i have more details i write more details here - from expierence to fix this problem

(0010181)
Teravus (administrator)
2009-03-28 06:09

I don't know :/

(0010182)
Jamenai (reporter)
2009-03-28 06:11

Ok ^^ but there are a lot of more people ^^

(0010183)
Ursula Matova (reporter)
2009-03-28 07:24

Just one idea, could be a Mono issue because I experienced such performance issues a long time ago.
Try first to upgrade mono ( 2.4 run fine for me on Linux Ubuntu 8.04 64bits )
Compiling from source is also a good idea.
Regards.

(0010184)
Teravus (administrator)
2009-03-28 07:32

I've seen these kind of bug reports get created, get ignored mostly, and then get uncerimoniously closed at a later date.

(0010185)
Ursula Matova (reporter)
2009-03-28 07:39

You're right Teravus ...
OpenSim run fine on Linux/64bits ... I haven't noticed any CPU issues these last months. The only thing is that the servers components use a bit more memory.

Other suggestions Jamenai,
- Try to upgrade your linux kernel ( that could have an impact )
- Try to setup the folowing environment variable before starting your OpenSim server :

export MONO_THREADS_PER_CPU=1000

Regards.

(0010186)
Jamenai (reporter)
2009-03-28 13:51

I have reinstalled mono and nant - >> downgraded to 1.9.1
- update mono and nant is impossible actually to 2.4 (distri probs)

After the reinstall and update to 8816 - result: It works ...actually...

Thanks...

PS: I change the status to resolved

(0010187)
Jamenai (reporter)
2009-03-28 13:53

I have reinstalled mono and nant - >> downgraded from mono 2.0 to 1.9.1

After the reinstall and update to 8816 - result: It works ...actually...

(0010559)
TerrorBite (reporter)
2009-04-09 04:52

I'm currently getting this on my opensim server, latest SVN revision 9055. Using Mono 1.9.1 I would get this issue once a day or so, the cli process would begin using 100% CPU and everything would stop responding. Now that I've upgraded to mono 2.4, I'm getting this every 10 to 15 minutes. I'm trying 'export MONO_THREADS_PER_CPU=1000' to see if this helps.

(0010560)
TerrorBite (reporter)
2009-04-09 05:06
edited on: 2009-04-09 05:06

export MONO_THREADS_PER_CPU=1000 didn't work, this run I got 100% CPU usage after about 10 minutes without even logging in. I might have to switch back to Mono 1.9.1 again.

(0010563)
TerrorBite (reporter)
2009-04-09 07:59

Reverting to Mono 1.9.1 has been unsucessful in reducing the frequency of the incidents (about 10 to 15 minutes after sim startup). While in the 100% CPU state, opensim freezes and does not respond to keystrokes on the console, and any logged-in clients disconenct from timeout. The opensim instance must then be killed and restarted.

(0010566)
LynMimistrobell (reporter)
2009-04-09 16:58

I'm running mono 1.9.1 on CentOS 32 bits, two instances of opensim. One of them shows this behaviour (I don't know after how long, but definately longer than an hour) the other one hasn't shown this yet.

(0010580)
TerrorBite (reporter)
2009-04-10 03:39

Here's a full thread dump (obtained by pressing Ctrl-\) that was taken while the mono process was at 99% CPU.

http://opensim.pastebin.com/m176f7c3 [^]

(0010581)
TerrorBite (reporter)
2009-04-10 05:02

I've disabled XEngine and switched to ScriptEngine.DotNetEngine, and this appears to have solved the problem. The XEngine backup process seems to be causing the issue.

(0010587)
TerrorBite (reporter)
2009-04-10 07:27

Problem wasn't solved after all, I now have both script engines disabled and I'm still getting this issue. This also rules out that any rogue scripts could be causing the high CPU load.

(0010601)
Jamenai (reporter)
2009-04-10 16:18

The problem comes back on mono 1.9.1
But i updated mono to 2.4 - works perfectly

(0010955)
M1sha (reporter)
2009-04-24 16:26

I've been experiencing this on the M1* regions. I'd tried CentOS and Debian with Mono 1.9.1, 2.4 and 2.5 (trunk) with no difference. After a chat with Nebadon I changed to Fedora fc10 plus Mono 2.4 from fc11 (rawhide). Curiously my Mono process is now idling at 0.0% instead of the 235.0% previously seen. I've no idea why this change helps, but changing to Fedora may help somebody else.

(0010985)
TerrorBite (reporter)
2009-04-25 14:04

RemedyTomm gave me a patch which I trialled, it seemed to fix the problem at first but after some testing it turns out that the bug still exists, but instead of after ten minutes it now occurs a number of hours after sim startup.

(0010988)
M1sha (reporter)
2009-04-25 16:03

Fedora turned out to be less successful than I thought. High CPU but working OK, was replaced by silently stopping.

(0010989)
RemedyTomm (reporter)
2009-04-25 16:07

Anyone wrestling with this, you might wanna give my patch at 0003511 a go. One of the symptoms of the issue I experienced was that the sim would rise to 100% cpu usage after a short time.

(0011221)
LynMimistrobell (reporter)
2009-05-01 07:45
edited on: 2009-05-01 21:05

I've noticed that the CPU load increase seems to occur when an avatar enters a region. It doesn't always happen when an avatar enters, but it seems that everytime the CPU load has gone up someone has entered the region... Maybe this is usefull to someone? :D

Any more tests done with Mono 2.4?

-- Update:

Today we updated the regions to 0.6.4.9362 and it seems to occur even more often now, and no longer related to agent entry. :(

Can we make this one a major please?

(0011657)
LenaVanilli (reporter)
2009-05-20 15:31
edited on: 2009-05-20 15:32

After weeks of struggling we solved our CPU-problem.

Server: SuSE 10.3, 64 bit, 8GB RAM, dual-core
OpenSim: 0.6.4.1-tag, UGAIM and 20 regions, all on the same server
Mono 2.4

We tried different values for MONO_THREADS_PER_CPU. We found out, that opensim runs most stable with the value of 300. By mistake we forgot to set the MONO_THREADS_PER_CPU. So the default took place. And ... it worked with the standard value of 10!

We made a stress-test using pcampbot.exe with 20 bots. It worked fine! Also the opensim runs without any breakdown now. Sometimes a bit laggy, but no more CPU-usage of 100%.

Some CPU usage statistics:

running 20 regions without any avatar loogged in: 10% - 15%
2 to 5 avatars logged in: 25% - 35%
after the FIRST logout of an avatar: 50% - 60%

(0011671)
LynMimistrobell (reporter)
2009-05-21 21:51

So, basically it will run fine with either 10 or 300 threads/cpu... but not something in between? Isn't that a bit weird?

I've upgraded to Mono 2.0.1 today (had to compile it myself, only 1.9 was available as binaries for CentOs 5) and I'm still seeing servercrashes after teleports. No MONO_THREADS_PER_CPU env variable set, so it should default to 10 - right?

(0011675)
LenaVanilli (reporter)
2009-05-21 22:53
edited on: 2009-05-21 22:54

We recommend to use 300 for MONO_THREADS_PER_CPU. If you have more than 4 Avatars logged in at the same time, its a must. Otherwise OpenSim will freeze.

I think you have to upgrade to 2.4 to avoid CPU problems. With this release we dont have any breakdowns.

We tested the mono 2.4 with OpenSim-Version 0.6.4.-post-fixes, and we faced the old problems with CPU usage up to 100%. Just try OpenSim 0.6.4.1 with mono 2.4!

(0011739)
Jamenai (reporter)
2009-05-26 07:02

...i mean it can be closed ( with mono 2.4. was the problem for me fixed ) ^^

(0011741)
LynMimistrobell (reporter)
2009-05-26 09:09

Last sunday I built mono 2.4, set MONO_THREADS_PER_CPU to 300. It seems to run a bit more stable, but every now and then an instance will still start using over 50% CPU and will continue to do so until restarted.

I am running the "stable" tagged SVN branch (r9683) - I hope to move to 0.6.5 this week.

(0012239)
Jamenai (reporter)
2009-06-22 08:49

The Problem was fixed with Mono Update 2.4
I closed this mantis

(0014942)
Enverex (reporter)
2010-02-07 15:19

I'm still randomly running into this issue. I've got no idea what causes it. The sim may be up for days, but then if anyone comes in and starts doing anything it may trigger the "CPU maxed out" bug.

CPU usage was at 0000030:0000015% prior to 2 people logging in. Shortly after they logged in and started building the machine shot to 800% CPU usage (4 core HT enabled Xeon X3450). Console is still responsive but the sim has to be shut down to get it back to normal CPU usage.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4926 gameserv 20 0 1036m 300m 22m S 800 7.6 2091:47 mono

Mono JIT compiler version 2.6.1 (tarball Sun Dec 20 16:04:39 CET 2009)
NAnt 0.86 (Build 0.86.2898.0; beta1; 08/12/2007)

OpenSim was built from GIT head about 5 days ago. Machine is running 64bit Arch Linux. 4GB RAM.

Let me know if you need any more information as this is making OpenSimulator unusable.

(0014943)
melanie (administrator)
2010-02-07 15:37

At this time, this is not fixable in a single place. There are numerous places where OpenSim is inefficient or hangs.

Intel are working hard on finding and resolving some of these issues.

(0014944)
Enverex (reporter)
2010-02-07 15:41

It may be worth adding that it seemed to be someone requesting Admin status that triggered it this time. I'll confirm that later when I'm back at my machine as I can't test it right now.

(0015392)
SnappyPappy (reporter)
2010-05-01 22:13

Seeing this pretty regularly as well.
OpenSim 0.6.8 PostFix-1
CentOS 5.4
Mono 2.4.2.3
4 CPUs, 4GB RAM
MONO_THREADS_PER_CPU=500
1 region running on this machine

We've been seeing this problem pretty regularly, but have not been able to reproduce it reliably (i.e. not specific set of steps to always reproduce). When watching the log, we'll start to notice WATCHDOG timeouts and we'll know the problem hit.

It happened to hit today and all 4 CPUs were being utilized at 100% by the mono process. One interesting note was that about 25% of that utilization on each CPU was in the kernel space.

Also, I looked at the threads that were running (using ps) and noticed that there were 119 threads running. All but 13 of them were created on or after the first WATCHDOG msg appeared. The sim was running for about 7 hours before this happpened.

I know it's nothing concrete, but hopefully will help point folks in the general direction.

(0017407)
Luisillo_Contepomi (reporter)
2010-11-28 16:41

I have overload cpu (4cpus) when opensim do statistic hour by hour...

when give in console...

Time now is 28/11/2010 14:40:20
Server has been running since Sunday, 28/11/2010 13:40:20
That is an elapsed time of 01:00:00.0293020
ASSET STATISTICS
bla bla...
CONNECTION STATISTICS
bla bla...
INVENTORY STATISTICS
bla bla...
FRAME STATISTICS
bla bla...
MEMORY STATISTICS
bla bla...
0 threads are being tracked:

*** ThreadPool threads ***
workers: 6 (500); ports: 0 (250)

when do this statistic... server go to 100% when finish statistic all come back to normal situation consuption about 10% cpu.

May be this can help.

(0018128)
Adamus (reporter)
2011-03-11 06:01
edited on: 2011-03-11 06:20

Running the following:

Ubuntu 10.10 Server
mono 2.6.7
16GB RAM
1 x Quad-core AMD CPU
opensim 0.7.0.2 (Grid Mode)

Found that XEngine causes mono spike CPU usage over 100% briefly when automatic script save is invoked. Actual spike value varies with sim load, but pattern is the same regardless of load. Parameter is "SaveInterval" in XEngine params. Default interval in OpenSim.ini is 120 seconds. Changed SaveInterval to other time values to test and sure enough, spike occurs at those intervals.
Currently running 4 regions, Grid mode, all regions running numerous scripts.
Not sure if it is a bug, or just a function of lots of scripts on the grid..

(0021479)
clairwil (reporter)
2012-05-22 18:18
edited on: 2012-05-22 18:21

Since upgrading to the new opensim software for OSG (OSgrid 0.7.4 (Dev) 0db60ee: 2012-05-20), I am also experiencing mono expansion as described above. I was running an older Mono; updated Mono to 2.10.9 Build 0000010 (latest stable release for Mac OS X). Issue continues with mono.

I would like to try to add the export MONO_THREADS_PER_CPU=100 to a .bashrc file to see if this fixes it. However, what is it in the opensim.ini file (or other files) what has changed that causes the issue? Seems like we should change a setting in the ini file instead of mono settings.

Is there an solution yet to the 100% CPU usage that can be accomplished within the opensim software settings?

I am running three sims on a Mac mini server in Lion and I have experienced no problems until this upgrade to opensim software. The machine slows to a crawl within 10 hours of running the sims and they must be restarted. Please let me know if you need additional documentation for an answer. Thanks.

(0021588)
Gwyneth Llewelyn (reporter)
2012-05-29 01:32

Seems to be related to http://opensimulator.org/mantis/view.php?id=6031 [^]

Allegedly MONO_THREADS_PER_CPU isn't supposed to be unnecessary under the latest versions of Mono (i.e. anything over 2.8, which is your case).

I have a similar problem, but it's not exactly the same as you have... CPU load under 0.7.3.1 is not really 100% per process, but overall load of the Mono processes has increased 10x since 0.7.2! Naturally, the server crawls to a stop... or almost. It does work indeed, just OpenSim is way slower than usual: some users report that terraforming is next to impossible to accomplish, while it used to be a breeze under 0.7.2 .

Notes
(0010178) Jamenai (reporter) 2009-03-28 05:58	Build 8916

(0010179) Teravus (administrator) 2009-03-28 06:01	I hate to say it, .. and I will. Unless you can provide a better bug report, chances are, this one will be ignored. It isn't a personal attack or anythin, it's just that with the information provided in this report, there's really nothing that we can do. We don't know enough about your system or the situation in which it occurs to be able to do anything about it.

(0010180) Jamenai (reporter) 2009-03-28 06:08 edited on: 2009-03-28 06:10	hi, if you need any information i can look what i can do. The last versions before 8862 are stable. If any other people have the same problem they can tell it and we get more informations. But its a bug or not? And if i have more details i write more details here - from expierence to fix this problem

(0010181) Teravus (administrator) 2009-03-28 06:09	I don't know :/

(0010182) Jamenai (reporter) 2009-03-28 06:11	Ok ^^ but there are a lot of more people ^^

(0010183) Ursula Matova (reporter) 2009-03-28 07:24	Just one idea, could be a Mono issue because I experienced such performance issues a long time ago. Try first to upgrade mono ( 2.4 run fine for me on Linux Ubuntu 8.04 64bits ) Compiling from source is also a good idea. Regards.

(0010184) Teravus (administrator) 2009-03-28 07:32	I've seen these kind of bug reports get created, get ignored mostly, and then get uncerimoniously closed at a later date.

(0010185) Ursula Matova (reporter) 2009-03-28 07:39	You're right Teravus ... OpenSim run fine on Linux/64bits ... I haven't noticed any CPU issues these last months. The only thing is that the servers components use a bit more memory. Other suggestions Jamenai, - Try to upgrade your linux kernel ( that could have an impact ) - Try to setup the folowing environment variable before starting your OpenSim server : export MONO_THREADS_PER_CPU=1000 Regards.

(0010186) Jamenai (reporter) 2009-03-28 13:51	I have reinstalled mono and nant - >> downgraded to 1.9.1 - update mono and nant is impossible actually to 2.4 (distri probs) After the reinstall and update to 8816 - result: It works ...actually... Thanks... PS: I change the status to resolved

(0010187) Jamenai (reporter) 2009-03-28 13:53	I have reinstalled mono and nant - >> downgraded from mono 2.0 to 1.9.1 After the reinstall and update to 8816 - result: It works ...actually...

(0010559) TerrorBite (reporter) 2009-04-09 04:52	I'm currently getting this on my opensim server, latest SVN revision 9055. Using Mono 1.9.1 I would get this issue once a day or so, the cli process would begin using 100% CPU and everything would stop responding. Now that I've upgraded to mono 2.4, I'm getting this every 10 to 15 minutes. I'm trying 'export MONO_THREADS_PER_CPU=1000' to see if this helps.

(0010560) TerrorBite (reporter) 2009-04-09 05:06 edited on: 2009-04-09 05:06	export MONO_THREADS_PER_CPU=1000 didn't work, this run I got 100% CPU usage after about 10 minutes without even logging in. I might have to switch back to Mono 1.9.1 again.

(0010563) TerrorBite (reporter) 2009-04-09 07:59	Reverting to Mono 1.9.1 has been unsucessful in reducing the frequency of the incidents (about 10 to 15 minutes after sim startup). While in the 100% CPU state, opensim freezes and does not respond to keystrokes on the console, and any logged-in clients disconenct from timeout. The opensim instance must then be killed and restarted.

(0010566) LynMimistrobell (reporter) 2009-04-09 16:58	I'm running mono 1.9.1 on CentOS 32 bits, two instances of opensim. One of them shows this behaviour (I don't know after how long, but definately longer than an hour) the other one hasn't shown this yet.

(0010580) TerrorBite (reporter) 2009-04-10 03:39	Here's a full thread dump (obtained by pressing Ctrl-\) that was taken while the mono process was at 99% CPU. http://opensim.pastebin.com/m176f7c3 [^]

(0010581) TerrorBite (reporter) 2009-04-10 05:02	I've disabled XEngine and switched to ScriptEngine.DotNetEngine, and this appears to have solved the problem. The XEngine backup process seems to be causing the issue.

(0010587) TerrorBite (reporter) 2009-04-10 07:27	Problem wasn't solved after all, I now have both script engines disabled and I'm still getting this issue. This also rules out that any rogue scripts could be causing the high CPU load.

(0010601) Jamenai (reporter) 2009-04-10 16:18	The problem comes back on mono 1.9.1 But i updated mono to 2.4 - works perfectly

(0010955) M1sha (reporter) 2009-04-24 16:26	I've been experiencing this on the M1* regions. I'd tried CentOS and Debian with Mono 1.9.1, 2.4 and 2.5 (trunk) with no difference. After a chat with Nebadon I changed to Fedora fc10 plus Mono 2.4 from fc11 (rawhide). Curiously my Mono process is now idling at 0.0% instead of the 235.0% previously seen. I've no idea why this change helps, but changing to Fedora may help somebody else.

(0010985) TerrorBite (reporter) 2009-04-25 14:04	RemedyTomm gave me a patch which I trialled, it seemed to fix the problem at first but after some testing it turns out that the bug still exists, but instead of after ten minutes it now occurs a number of hours after sim startup.

(0010988) M1sha (reporter) 2009-04-25 16:03	Fedora turned out to be less successful than I thought. High CPU but working OK, was replaced by silently stopping.

(0010989) RemedyTomm (reporter) 2009-04-25 16:07	Anyone wrestling with this, you might wanna give my patch at 0003511 a go. One of the symptoms of the issue I experienced was that the sim would rise to 100% cpu usage after a short time.

(0011221) LynMimistrobell (reporter) 2009-05-01 07:45 edited on: 2009-05-01 21:05	I've noticed that the CPU load increase seems to occur when an avatar enters a region. It doesn't always happen when an avatar enters, but it seems that everytime the CPU load has gone up someone has entered the region... Maybe this is usefull to someone? :D Any more tests done with Mono 2.4? -- Update: Today we updated the regions to 0.6.4.9362 and it seems to occur even more often now, and no longer related to agent entry. :( Can we make this one a major please?

(0011657) LenaVanilli (reporter) 2009-05-20 15:31 edited on: 2009-05-20 15:32	After weeks of struggling we solved our CPU-problem. Server: SuSE 10.3, 64 bit, 8GB RAM, dual-core OpenSim: 0.6.4.1-tag, UGAIM and 20 regions, all on the same server Mono 2.4 We tried different values for MONO_THREADS_PER_CPU. We found out, that opensim runs most stable with the value of 300. By mistake we forgot to set the MONO_THREADS_PER_CPU. So the default took place. And ... it worked with the standard value of 10! We made a stress-test using pcampbot.exe with 20 bots. It worked fine! Also the opensim runs without any breakdown now. Sometimes a bit laggy, but no more CPU-usage of 100%. Some CPU usage statistics: running 20 regions without any avatar loogged in: 10% - 15% 2 to 5 avatars logged in: 25% - 35% after the FIRST logout of an avatar: 50% - 60%

(0011671) LynMimistrobell (reporter) 2009-05-21 21:51	So, basically it will run fine with either 10 or 300 threads/cpu... but not something in between? Isn't that a bit weird? I've upgraded to Mono 2.0.1 today (had to compile it myself, only 1.9 was available as binaries for CentOs 5) and I'm still seeing servercrashes after teleports. No MONO_THREADS_PER_CPU env variable set, so it should default to 10 - right?

(0011675) LenaVanilli (reporter) 2009-05-21 22:53 edited on: 2009-05-21 22:54	We recommend to use 300 for MONO_THREADS_PER_CPU. If you have more than 4 Avatars logged in at the same time, its a must. Otherwise OpenSim will freeze. I think you have to upgrade to 2.4 to avoid CPU problems. With this release we dont have any breakdowns. We tested the mono 2.4 with OpenSim-Version 0.6.4.-post-fixes, and we faced the old problems with CPU usage up to 100%. Just try OpenSim 0.6.4.1 with mono 2.4!

(0011739) Jamenai (reporter) 2009-05-26 07:02	...i mean it can be closed ( with mono 2.4. was the problem for me fixed ) ^^

(0011741) LynMimistrobell (reporter) 2009-05-26 09:09	Last sunday I built mono 2.4, set MONO_THREADS_PER_CPU to 300. It seems to run a bit more stable, but every now and then an instance will still start using over 50% CPU and will continue to do so until restarted. I am running the "stable" tagged SVN branch (r9683) - I hope to move to 0.6.5 this week.

(0012239) Jamenai (reporter) 2009-06-22 08:49	The Problem was fixed with Mono Update 2.4 I closed this mantis

(0014942) Enverex (reporter) 2010-02-07 15:19	I'm still randomly running into this issue. I've got no idea what causes it. The sim may be up for days, but then if anyone comes in and starts doing anything it may trigger the "CPU maxed out" bug. CPU usage was at 0000030:0000015% prior to 2 people logging in. Shortly after they logged in and started building the machine shot to 800% CPU usage (4 core HT enabled Xeon X3450). Console is still responsive but the sim has to be shut down to get it back to normal CPU usage. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4926 gameserv 20 0 1036m 300m 22m S 800 7.6 2091:47 mono Mono JIT compiler version 2.6.1 (tarball Sun Dec 20 16:04:39 CET 2009) NAnt 0.86 (Build 0.86.2898.0; beta1; 08/12/2007) OpenSim was built from GIT head about 5 days ago. Machine is running 64bit Arch Linux. 4GB RAM. Let me know if you need any more information as this is making OpenSimulator unusable.

(0014943) melanie (administrator) 2010-02-07 15:37	At this time, this is not fixable in a single place. There are numerous places where OpenSim is inefficient or hangs. Intel are working hard on finding and resolving some of these issues.

(0014944) Enverex (reporter) 2010-02-07 15:41	It may be worth adding that it seemed to be someone requesting Admin status that triggered it this time. I'll confirm that later when I'm back at my machine as I can't test it right now.

(0015392) SnappyPappy (reporter) 2010-05-01 22:13	Seeing this pretty regularly as well. OpenSim 0.6.8 PostFix-1 CentOS 5.4 Mono 2.4.2.3 4 CPUs, 4GB RAM MONO_THREADS_PER_CPU=500 1 region running on this machine We've been seeing this problem pretty regularly, but have not been able to reproduce it reliably (i.e. not specific set of steps to always reproduce). When watching the log, we'll start to notice WATCHDOG timeouts and we'll know the problem hit. It happened to hit today and all 4 CPUs were being utilized at 100% by the mono process. One interesting note was that about 25% of that utilization on each CPU was in the kernel space. Also, I looked at the threads that were running (using ps) and noticed that there were 119 threads running. All but 13 of them were created on or after the first WATCHDOG msg appeared. The sim was running for about 7 hours before this happpened. I know it's nothing concrete, but hopefully will help point folks in the general direction.

(0017407) Luisillo_Contepomi (reporter) 2010-11-28 16:41	I have overload cpu (4cpus) when opensim do statistic hour by hour... when give in console... Time now is 28/11/2010 14:40:20 Server has been running since Sunday, 28/11/2010 13:40:20 That is an elapsed time of 01:00:00.0293020 ASSET STATISTICS bla bla... CONNECTION STATISTICS bla bla... INVENTORY STATISTICS bla bla... FRAME STATISTICS bla bla... MEMORY STATISTICS bla bla... 0 threads are being tracked: * ThreadPool threads * workers: 6 (500); ports: 0 (250) when do this statistic... server go to 100% when finish statistic all come back to normal situation consuption about 10% cpu. May be this can help.

(0018128) Adamus (reporter) 2011-03-11 06:01 edited on: 2011-03-11 06:20	Running the following: Ubuntu 10.10 Server mono 2.6.7 16GB RAM 1 x Quad-core AMD CPU opensim 0.7.0.2 (Grid Mode) Found that XEngine causes mono spike CPU usage over 100% briefly when automatic script save is invoked. Actual spike value varies with sim load, but pattern is the same regardless of load. Parameter is "SaveInterval" in XEngine params. Default interval in OpenSim.ini is 120 seconds. Changed SaveInterval to other time values to test and sure enough, spike occurs at those intervals. Currently running 4 regions, Grid mode, all regions running numerous scripts. Not sure if it is a bug, or just a function of lots of scripts on the grid..

(0021479) clairwil (reporter) 2012-05-22 18:18 edited on: 2012-05-22 18:21	Since upgrading to the new opensim software for OSG (OSgrid 0.7.4 (Dev) 0db60ee: 2012-05-20), I am also experiencing mono expansion as described above. I was running an older Mono; updated Mono to 2.10.9 Build 0000010 (latest stable release for Mac OS X). Issue continues with mono. I would like to try to add the export MONO_THREADS_PER_CPU=100 to a .bashrc file to see if this fixes it. However, what is it in the opensim.ini file (or other files) what has changed that causes the issue? Seems like we should change a setting in the ini file instead of mono settings. Is there an solution yet to the 100% CPU usage that can be accomplished within the opensim software settings? I am running three sims on a Mac mini server in Lion and I have experienced no problems until this upgrade to opensim software. The machine slows to a crawl within 10 hours of running the sims and they must be restarted. Please let me know if you need additional documentation for an answer. Thanks.

(0021588) Gwyneth Llewelyn (reporter) 2012-05-29 01:32	Seems to be related to http://opensimulator.org/mantis/view.php?id=6031 [^] Allegedly MONO_THREADS_PER_CPU isn't supposed to be unnecessary under the latest versions of Mono (i.e. anything over 2.8, which is your case). I have a similar problem, but it's not exactly the same as you have... CPU load under 0.7.3.1 is not really 100% per process, but overall load of the Mono processes has increased 10x since 0.7.2! Naturally, the server crawls to a stop... or almost. It does work indeed, just OpenSim is way slower than usual: some users report that terraforming is next to impossible to accomplish, while it used to be a breeze under 0.7.2 .

Issue History
Date Modified	Username	Field	Change
2009-03-28 05:57	Jamenai	New Issue
2009-03-28 05:57	Jamenai	SVN Revision	=> 0
2009-03-28 05:57	Jamenai	Run Mode	=> Grid (Multiple Regions per Sim)
2009-03-28 05:57	Jamenai	Physics Engine	=> ODE
2009-03-28 05:57	Jamenai	Environment	=> Mono / Linux64
2009-03-28 05:57	Jamenai	Mono Version	=> 2.0.0
2009-03-28 05:58	Jamenai	Note Added: 0010178
2009-03-28 06:01	Teravus	Note Added: 0010179
2009-03-28 06:08	Jamenai	Note Added: 0010180
2009-03-28 06:08	Jamenai	Note Edited: 0010180
2009-03-28 06:09	Jamenai	Note Edited: 0010180
2009-03-28 06:09	Teravus	Note Added: 0010181
2009-03-28 06:10	Jamenai	Note Edited: 0010180
2009-03-28 06:11	Jamenai	Note Added: 0010182
2009-03-28 07:24	Ursula Matova	Note Added: 0010183
2009-03-28 07:32	Teravus	Note Added: 0010184
2009-03-28 07:39	Ursula Matova	Note Added: 0010185
2009-03-28 13:51	Jamenai	Note Added: 0010186
2009-03-28 13:53	Jamenai	Mono Version	2.0.0 => 1.9.1
2009-03-28 13:53	Jamenai	Status	new => resolved
2009-03-28 13:53	Jamenai	Resolution	open => fixed
2009-03-28 13:53	Jamenai	Assigned To	=> Jamenai
2009-03-28 13:53	Jamenai	Note Added: 0010187
2009-04-09 04:52	TerrorBite	Status	resolved => feedback
2009-04-09 04:52	TerrorBite	Resolution	fixed => reopened
2009-04-09 04:52	TerrorBite	Note Added: 0010559
2009-04-09 05:06	TerrorBite	Note Added: 0010560
2009-04-09 05:06	TerrorBite	Note Edited: 0010560
2009-04-09 07:59	TerrorBite	Note Added: 0010563
2009-04-09 16:58	LynMimistrobell	Note Added: 0010566
2009-04-10 03:39	TerrorBite	Note Added: 0010580
2009-04-10 05:02	TerrorBite	Note Added: 0010581
2009-04-10 07:27	TerrorBite	Note Added: 0010587
2009-04-10 16:18	Jamenai	Note Added: 0010601
2009-04-24 16:26	M1sha	Note Added: 0010955
2009-04-25 14:04	TerrorBite	Note Added: 0010985
2009-04-25 16:03	M1sha	Note Added: 0010988
2009-04-25 16:07	RemedyTomm	Note Added: 0010989
2009-05-01 07:45	LynMimistrobell	Note Added: 0011221
2009-05-01 21:05	LynMimistrobell	Note Edited: 0011221
2009-05-20 15:31	LenaVanilli	Note Added: 0011657
2009-05-20 15:32	LenaVanilli	Note Edited: 0011657
2009-05-21 21:51	LynMimistrobell	Note Added: 0011671
2009-05-21 22:53	LenaVanilli	Note Added: 0011675
2009-05-21 22:54	LenaVanilli	Note Edited: 0011675
2009-05-26 07:02	Jamenai	Note Added: 0011739
2009-05-26 09:09	LynMimistrobell	Note Added: 0011741
2009-06-22 08:49	Jamenai	Mono Version	1.9.1 => Other
2009-06-22 08:49	Jamenai	Status	feedback => resolved
2009-06-22 08:49	Jamenai	Resolution	reopened => fixed
2009-06-22 08:49	Jamenai	Note Added: 0012239
2009-07-08 14:59	chi11ken	Status	resolved => closed
2010-02-07 15:19	Enverex	Status	closed => feedback
2010-02-07 15:19	Enverex	Resolution	fixed => reopened
2010-02-07 15:19	Enverex	Note Added: 0014942
2010-02-07 15:37	melanie	Note Added: 0014943
2010-02-07 15:41	Enverex	Note Added: 0014944
2010-05-01 22:13	SnappyPappy	Note Added: 0015392
2010-11-28 16:41	Luisillo_Contepomi	Note Added: 0017407
2011-03-11 06:01	Adamus	Note Added: 0018128
2011-03-11 06:10	Adamus	Note Edited: 0018128
2011-03-11 06:18	Adamus	Note Edited: 0018128
2011-03-11 06:20	Adamus	Note Edited: 0018128
2012-05-22 18:18	clairwil	Note Added: 0021479
2012-05-22 18:21	clairwil	Note Edited: 0021479	View Revisions
2012-05-29 01:32	Gwyneth Llewelyn	Note Added: 0021588
2012-05-29 01:33	Gwyneth Llewelyn	Relationship added	related to 0006031