Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006030opensim[REGION] OpenSim Corepublic2012-05-23 10:192014-02-07 15:25
Reporterclairwil 
Assigned To 
PriorityhighSeveritymajorReproducibilityalways
StatusnewResolutionopen 
PlatformMacOSLionOS Version
Product Version 
Target VersionFixed in Version 
Summary0006030: Mono resource allocation grows rapidly while running opensim until CPU cannot function
DescriptionI upgraded three sims to opensim v 0.7.4. All three had been stable in v 0.7.3 showing mono at about 250 in the Mac Activity Monitor. After upgrading, they doubled to 500 within minutes of start up and within 8 hours were at more than 1000.
Steps To ReproduceRestarting the computer and sims showed the same uptick with mono. It was suggested that this was related to scripts in the sims. Therefore, I did a test. I set up three sims:

1. Sim One, running as before in v 0.7.3 (two regions only but TONS of scripts).
2. Sim Two, running v 0.7.4 with all scripts disabled.
3. Sim Three, running v 0.7.4 with all scripts ON. (my control)

I have posted screenshots on my flickr site showing the activity monitor, with descriptions of how the sims ran with mono. here's a url: http://www.flickr.com/photos/clairwiloh/ [^]

I am uploading one image, a screenshot from the Mac Activity Monitor, showing the three runs of mono after running these three test sims overnight.

The short version is that the sim running in v 0.7.3 continued to show up at about 250 in the monitor. The other two jumped up to about 500 and, over time, ran to around 1000. The one with scripts disabled used slightly less resources than the one with scripts on. But both sims in v 0.7.4 are showing this same memory resource increase.
Additional InformationI need to add that, when I saw the issue was not fixed by a restart, my first thought was to check mono. I updated to the newest stable mono for Mac. The problem did not resolve. After more testing with the three sims set as described above, I did one more mono upgrade; I upgraded to the latest beta mono. It continues to run smoothly but with the same memory issues. So, switching mono versions changed nothing.

it has been suggested that I use a .bashrc file with the command

export MONO_THREADS_PER_CPU=100

to hold down mono. I did try to set this up, but I am not that familiar with configuring mono with external config files and I don't think I executed this correctly.

I think that the best solution would be within opensim.ini rather than an external tweak to mono.

Thanks for checking this mantis. I am ready to do any tests suggested to troubleshoot this issue.
TagsNo tags attached.
Git Revision or version number0db60eea85f16b0b428190590b8e8ca1392a2e35 : [r/19080]
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBasicPhysics
Script Engine
EnvironmentMono / OSX
Mono Version2.10
Viewervarious.
Attached Filesjpg file icon mono_running_three_sims.jpg [^] (157,300 bytes) 2012-05-23 10:19

- Relationships
related to 0006031closedkenvc 'System.OutOfMemoryException' - Simulators are consuming much more memory than in the past. 
related to 0007002closedmelanie Use signaling instead of CPU-blocking sleep for LSL events (listen, timer, sensor and dataserver). 

-  Notes
(0021505)
Taarna Welles (reporter)
2012-05-23 13:30

I can confirm this increased memory usage in the latest versions of opensim since I monitor the memory use in the taskmanager of windows 2008. We thought it maybe related to more scripts and regions but disabling those showed still increased memory usage. Please contact me for more details.

Kind regards,

Taarna Welles

taarna.welles@ziggo.nl
(0021517)
clairwil (reporter)
2012-05-23 16:49

This may be premature, but today someone suggested that I comment out this line in the ini file:

MapImageModule = "Warp3DImageModule"

Others had reported that this was causing memory use to climb, especially when many regions were being served. Since I have only 12 regions, I did not expect to see a big change but I commented out the line and restarted the two test sims running opensim version 0.7.4.

In previous runs of the software, memory usage climbed to about 500-550 within two minutes of runtime. I am ten minutes into runtime and the memory usage is sitting on 250 and not climbing. I'll come back in a few hours to report whether this held steady. Feeling optimistic. Interesting that this was more extreme on the Mac platform than on the Windows platform (if indeed this is the only issue).

Much thanks to AradTechKK for this suggestion (and to Richard Raymaker, who seconded this opinion). The IRC is a wonderful source of support, especially for those of us who are more tinkerers than techies).
(0021521)
justincc (administrator)
2012-05-23 20:01

This might be associated with periodically generating new warp 3d map tiles (via MaptileRefresh being something other than 0). Suggest also explicitly setting this to 0.
(0021523)
clairwil (reporter)
2012-05-24 07:50

thanks, Justin, I'll consider resetting it and testing again. Happy to report that the situation appears completely resolved by commenting out MapImageModule.
(0021524)
BlueWall (administrator)
2012-05-24 08:07

Change category from DSG to Core
(0021567)
Gwyneth Llewelyn (reporter)
2012-05-28 03:48

Hmm. On 0.7.3.1, I have explicitly set:

MapImageModule = "MapImageModule"

and never tested the Warp3D module.

My map tile configuration is:

    ;; Map tile options. You can choose to generate no map tiles at all,
    ;; generate normal maptiles, or nominate an uploaded texture to
    ;; be the map tile
    GenerateMaptiles = true

    ;; If desired, a running region can update the map tiles periodically
    ;; to reflect building activity. This names no sense of you don't have
    ;; prims on maptiles. Value is in seconds.
    ; MaptileRefresh = 0

    ;; If not generating maptiles, use this static texture asset ID
    ; MaptileStaticUUID = "00000000-0000-0000-0000-000000000000"

    ;# {TextureOnMapTile} {} {Use terrain textures for map tiles?} {true false} true
    ;; Use terrain texture for maptiles if true, use shaded green if false
    TextureOnMapTile = true

    ;# {DrawPrimOnMapTile} {} {Draw prim shapes on map tiles?} {true false} false
    ;; Draw objects on maptile. This step might take a long time if you've
    ;; got a large number of objects, so you can turn it off here if you'd like.
    DrawPrimOnMapTile = true

I wonder if shutting down DrawPrimOnMapTile will lower memory requirements...
(0021605)
Gwyneth Llewelyn (reporter)
2012-05-31 16:01

Ok, I've commented out MapImageModule as well. There is an improvement in server performance, even with avatars logged in. And the map still shows the map tiles... what does "MapImageModule" actually do? I've "recovered" that option from an ancient configuration... it seems not be used any more?
(0021609)
justincc (administrator)
2012-05-31 21:06

Map image regenerates map tiles on restart, though there are also settings to do this periodically.

OpenSim would reuse the last maptile generated which is why you're still seeing these I expect. The tile mechanism is also different between v1 and v2/v3 viewers, which use entirely different approaches to fetch the data (v1 fetches from the sim itself whilst v2/v3 go direct to set a of tiles in the bin/maptiles directory of the robust machine, I believe.
(0021626)
justincc (administrator)
2012-06-05 20:26

You may want to try git master code as of commit 514dd85. This makes sure all Bitmaps used by map image generation are explicitly Dispose()d but I don't know if this will help.
(0021963)
CrasherRob (reporter)
2012-08-02 09:27

This problem still recurs as of release 19933, although the memory usage build-up might be slower. On Windows, the memory build-up can be measured via the task manager. As long as it stays under whichever (physical or virtual) memory limit installation maximum you have, there doesn't seem to be any real adverse effect.

- Issue History
Date Modified Username Field Change
2012-05-23 10:19 clairwil New Issue
2012-05-23 10:19 clairwil File Added: mono_running_three_sims.jpg
2012-05-23 13:30 Taarna Welles Note Added: 0021505
2012-05-23 16:49 clairwil Note Added: 0021517
2012-05-23 20:01 justincc Note Added: 0021521
2012-05-24 07:50 clairwil Note Added: 0021523
2012-05-24 08:07 BlueWall Note Added: 0021524
2012-05-24 08:07 BlueWall Category [MISC] DSG => [REGION] OpenSim Core
2012-05-28 03:48 Gwyneth Llewelyn Note Added: 0021567
2012-05-31 16:01 Gwyneth Llewelyn Note Added: 0021605
2012-05-31 16:13 Gwyneth Llewelyn Relationship added related to 0006031
2012-05-31 21:06 justincc Note Added: 0021609
2012-06-05 20:26 justincc Note Added: 0021626
2012-08-02 09:27 CrasherRob Note Added: 0021963
2014-02-07 15:25 user2213 Relationship added related to 0007002


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker