Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008177opensim[REGION] OpenSim Corepublic2017-05-23 13:112017-05-27 23:30
Reporterwatcher64 
Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
StatusnewResolutionopen 
PlatformLINUXOSUBUNTU x64OS Version14/15/16
Product Versionmaster (dev code) 
Target VersionFixed in Version 
Summary0008177: Extreme memory usage as compared to 0.8.2.1 and 0.8.3 Post-Fixes
DescriptionI have tested this by loading up a region in 8x using bullet physics.

When loading up the exact same region in the .9 master code the same region uses over double the amount of RAM regardless of using bullet or ubODE physics.
regions are 4x4 var.


on 8x region using approx 2700mb of ram
on 9x region using approx 5600mb of ram

This is not a leak , this is right after start up and logins enable.
Steps To ReproduceI have tested this by loading up a region in 8x using bullet physics.

When loading up the exact same region in the .9 master code the same region uses over double the amount of RAM regardless of using bullet or ubODE physics.
regions are 4x4 var.


on 8x region using approx 2700mb of ram
on 9x region using approx 5600mb of ram

This is not a leak , this is right after start up and logins enable.
Additional InformationI have not noticed this previously , it could have been there for a while .

TagsNo tags attached.
Git Revision or version number
Run Mode Grid (1 Region per Sim) , Grid (Multiple Regions per Sim)
Physics EngineBulletSim, ubODE
EnvironmentMono / Linux64
Mono VersionOther
ViewerANY
Attached Files

- Relationships

-  Notes
(0031923)
kcozens (administrator)
2017-05-23 13:24

Which version of mono are you using?
(0031924)
watcher64 (reporter)
2017-05-23 14:07

I noticed with 4.2 , then with 4.8 and now my servers evidently auto updated to 5 ..
(0031925)
watcher64 (reporter)
2017-05-23 15:09

I am going to test on my backup sever on mono 3.2.8 , see if that does the same thing. ..
(0031927)
watcher64 (reporter)
2017-05-23 16:26
edited on: 2017-05-23 16:33

Now tested on mono 3.2.8, and above this is just a random sampling, but same oar loaded .. (this is right after startup on .9x and a few hours later on 8.x)

On 9.x Process memory: Physical 4469 MB

On 8.x Process memory : 2430 MB

I have tried on the .9x with both bullet and ubODE makes no noticeable difference.


And just for informational purposes, on either system does it appear that it is using much cpu, which is pretty normal for opensim, but on the .9x it is about 50% higher, which is still not that much.

Oh and just to verify I am testing with one region per instance.

As stated before this is a 4x4 var, with about 17 thousand objects ..

On the .9x it will spike sometimes and when it does you can hardly move at all.

(0031932)
watcher64 (reporter)
2017-05-23 18:16

ok both sims tested on 5 different versions of mono , difference between versions is negligible.

Sims have been up for an hour, started at the same time, with clean reboot of the server ..

Show stats from each.

.8x
MEMORY STATISTICS
Heap allocated to OpenSim : 672 MB
Last heap allocation rate : 0.027 MB/s
Average heap allocation rate: 0.13 MB/s
Process memory : 2377 MBMEMORY STATISTICS

.9x
MEMORY STATISTICS
Heap allocated to OpenSim : 3152 MB
Last heap allocation rate : 0.145 MB/s
Average heap allocation rate: 0.038 MB/s
Process memory: Physical 5469 MB Paged 0 MB Virtual 8798 MB
Peak process memory: Physical 5470 MB Paged 0 MB Virtual 8798 MB


I think this is a little excessive ..
(0031933)
UbitUmarov (administrator)
2017-05-23 18:27

just a bit..
and total opposite to what I seen in win7 now
(0031942)
watcher64 (reporter)
2017-05-23 18:51

Yes and as I said those sims are exact copies of each other , with NO avatars or NPC's on them .. A lot of scripts and objects , but no agents ..
(0031965)
UbitUmarov (administrator)
2017-05-25 22:46

testing with mono 5.0 with Bill

starting region with the opensim-ode.sh as
--
ulimit -s 262144 <-- for ode and ubdoe
MONO_GC_PARAMS="minor=split,promotion-age=14"
mono --desktop OpenSim.exe
---

does have big impact on mono memory use
bill is testing other option, he may report
(0031966)
watcher64 (reporter)
2017-05-25 22:53

If you guys have not figured it out yet I am Bill Blight ..


I am testing with some overly complicated settings that seem to have a better result on a region with 33k prims and 2700 scripts, but will test longer term and report back ..

The settings I am using are ..

ulimit -s 1048576
export MONO_GC_PARAMS="nursery-size=16m,evacuation-threshold=90,soft-heap-limit=1280m,save-target-ratio=0.1,default-allowance-ratio=1.0,promotion-age=14,minor=split,major=marksweep,no-lazy-sweep,alloc-ratio=50"
export MONO_GC_DEBUG="xdomain-checks,clear-at-gc"
export MONO_ENV_OPTIONS="--desktop"
export MONO_THREADS_PER_CPU=250


These do appear to slow down region startup for some reason , but after the region is up it seems to run just fine, with some better memory management.
(0031970)
UbitUmarov (administrator)
2017-05-26 15:27

Compile as Release does help a bit on all aspects

 Just if something goes wrong, there is less information to debug it
(0031972)
watcher64 (reporter)
2017-05-27 19:39

Well let's see how can I put this, umm , still crap ..

33k objects 2700 scripts

Region on .8x running for 24 hours

MEMORY STATISTICS
Heap allocated to OpenSim : 751 MB
Last heap allocation rate : 2.048 MB/s
Average heap allocation rate: 0.914 MB/s
Process memory : 3514 MB

Runs fine , no lag what so ever ..

.9x up for just over an hour ..
MEMORY STATISTICS
Heap allocated to OpenSim : 1399 MB
Last heap allocation rate : 0 MB/s
Average heap allocation rate: 0 MB/s
Process memory: Physical 4452 MB Paged 0 MB Virtual 7780 MB
Peak process memory: Physical 4452 MB Paged 0 MB Virtual 7781 MB

Can hardly move on the region ..
(0031973)
UbitUmarov (administrator)
2017-05-27 20:05

crude win test,just after logins enable, no avatar log in

region 0.82, bullet 2x size
MEMORY STATISTICS
Heap allocated to OpenSim : 54 MB
Last heap allocation rate : 0.016 MB/s
Average heap allocation rate: 0.015 MB/s
Process memory : 376 MB

0.91 region 3x + 1x on same instance. 1x has same prims as above, 3x has a few more
MEMORY STATISTICS
Heap allocated to OpenSim : 58 MB
Last heap allocation rate : 0.104 MB/s
Average heap allocation rate: 0.104 MB/s
Process memory: Physical 171 MB Paged 152 MB Virtual 740 MB
Peak process memory: Physical 267 MB Paged 249 MB Virtual 763 MB
(0031974)
watcher64 (reporter)
2017-05-27 20:07

yeah yeah yeah .... Win win win .. :P

Who's gonna pay for my windows licenses?

But I guess the issue is .. .8x does not have this memory issue as .9x does on linux ...

What is the root cause of .9x using more memory on linux?
(0031975)
UbitUmarov (administrator)
2017-05-27 20:11

driving same car on identical terrain, both regions on bullet
0.91 150ms avg physics time
0.82 320ms avg physics time ( but ok 0.82 stats was bad )

with the avatar log and a few laps with car
0.82
MEMORY STATISTICS
Heap allocated to OpenSim : 91 MB
Last heap allocation rate : 0.034 MB/s
Average heap allocation rate: 0.05 MB/s
Process memory : 405 MB

0.91
MEMORY STATISTICS
Heap allocated to OpenSim : 64 MB
Last heap allocation rate : 0.111 MB/s
Average heap allocation rate: 0.104 MB/s
Process memory: Physical 187 MB Paged 163 MB Virtual 746 MB
Peak process memory: Physical 267 MB Paged 249 MB Virtual 771 MB
(0031976)
UbitUmarov (administrator)
2017-05-27 20:23

avatar login with clear cache in similar position, full inv fetch objects and textures

0.82
MEMORY STATISTICS
Heap allocated to OpenSim : 81 MB
Last heap allocation rate : 0.054 MB/s
Average heap allocation rate: 4.028 MB/s
Process memory : 428 MB

0.91
MEMORY STATISTICS
Heap allocated to OpenSim : 68 MB
Last heap allocation rate : 3.524 MB/s
Average heap allocation rate: 3.318 MB/s
Process memory: Physical 210 MB Paged 219 MB Virtual 756 MB
Peak process memory: Physical 306 MB Paged 288 MB Virtual 779 MB

after avatar log out

0.82
MEMORY STATISTICS
Heap allocated to OpenSim : 96 MB
Last heap allocation rate : 0.009 MB/s
Average heap allocation rate: 0.045 MB/s
Process memory : 429 MB

0.91
MEMORY STATISTICS
Heap allocated to OpenSim : 64 MB
Last heap allocation rate : 0.125 MB/s
Average heap allocation rate: 0.107 MB/s
Process memory: Physical 195 MB Paged 203 MB Virtual 746 MB
Peak process memory: Physical 306 MB Paged 288 MB Virtual 779 MB

so not sure what to say :(
(0031977)
UbitUmarov (administrator)
2017-05-27 20:28

after avatar return ( kicks gc sometimes)

0.82
MEMORY STATISTICS
Heap allocated to OpenSim : 101 MB
Last heap allocation rate : 0.048 MB/s
Average heap allocation rate: 0.78 MB/s
Process memory : 433 MB

0.91
MEMORY STATISTICS
Heap allocated to OpenSim : 61 MB
Last heap allocation rate : 0.098 MB/s
Average heap allocation rate: 0.108 MB/s
Process memory: Physical 183 MB Paged 164 MB Virtual 745 MB
Peak process memory: Physical 306 MB Paged 288 MB Virtual 779 MB
(0031978)
watcher64 (reporter)
2017-05-27 20:28

So essentially we are just screwed if we want to continue running on linux , we are stuck with .8x ...
(0031979)
UbitUmarov (administrator)
2017-05-27 20:30

versions where
urrent simulator: OpenSim 0.9.1.0 Dev (Win/.NET) ( master )
Previous simulator: OpenSim 0.8.2.1 Post_Fixes (Win/.NET)]
(0031980)
UbitUmarov (administrator)
2017-05-27 21:20

your VM, mono5, 1x region, identical oar. ubOde
started with
ulimit -s 262144 <-- for ode and ubdoe
MONO_GC_PARAMS="minor=split,promotion-age=14"
mono --desktop OpenSim.exe


MEMORY STATISTICS
Heap allocated to OpenSim : 40 MB
Last heap allocation rate : 0 MB/s
Average heap allocation rate: 0.002 MB/s
Process memory: Physical 189 MB Paged 0 MB Virtual 4710 MB
Peak process memory: Physical 273 MB Paged 0 MB Virtual 4788 MB

same win 0.91 as before but also on ubOde now
MEMORY STATISTICS
Heap allocated to OpenSim : 52 MB
Last heap allocation rate : 0.094 MB/s
Average heap allocation rate: 0.102 MB/s
Process memory: Physical 169 MB Paged 143 MB Virtual 769 MB
Peak process memory: Physical 262 MB Paged 239 MB Virtual 786 MB
(0031981)
UbitUmarov (administrator)
2017-05-27 22:22

sorry about car timings where us not ms
(0031982)
UbitUmarov (administrator)
2017-05-27 23:30

before going to sleep
did some crossings and little prims rez on 0.91 regions above, both win and mono on ubode, stats as I type

Linux VM mono 5
EMORY STATISTICS
Heap allocated to OpenSim : 46 MB
Last heap allocation rate : 0 MB/s
Average heap allocation rate: 0.004 MB/s
Process memory: Physical 250 MB Paged 0 MB Virtual 4757 MB
Peak process memory: Physical 300 MB Paged 0 MB Virtual 4808 MB

win 91 2 regions as above
EMORY STATISTICS
eap allocated to OpenSim : 50 MB
ast heap allocation rate : 0.113 MB/s
verage heap allocation rate: 0.112 MB/s
rocess memory: Physical 185 MB Paged 160 MB Virtual 779 MB
eak process memory: Physical 346 MB Paged 320 MB Virtual 920 MB

win 0.82 just being idle since last stats repo above
MEMORY STATISTICS
Heap allocated to OpenSim : 272 MB
Last heap allocation rate : 0.015 MB/s
Average heap allocation rate: 0.045 MB/s
Process memory : 521 MB

so 0.82 is great? well maybe.. this is just idle test

- Issue History
Date Modified Username Field Change
2017-05-23 13:11 watcher64 New Issue
2017-05-23 13:21 watcher64 Category [MISC] Compiling / Building => [REGION] OpenSim Core
2017-05-23 13:24 kcozens Note Added: 0031923
2017-05-23 14:07 watcher64 Note Added: 0031924
2017-05-23 15:09 watcher64 Note Added: 0031925
2017-05-23 16:26 watcher64 Note Added: 0031927
2017-05-23 16:29 watcher64 Note Edited: 0031927 View Revisions
2017-05-23 16:31 watcher64 Note Edited: 0031927 View Revisions
2017-05-23 16:33 watcher64 Note Edited: 0031927 View Revisions
2017-05-23 18:16 watcher64 Note Added: 0031932
2017-05-23 18:27 UbitUmarov Note Added: 0031933
2017-05-23 18:51 watcher64 Note Added: 0031942
2017-05-25 22:46 UbitUmarov Note Added: 0031965
2017-05-25 22:53 watcher64 Note Added: 0031966
2017-05-26 15:27 UbitUmarov Note Added: 0031970
2017-05-27 19:39 watcher64 Note Added: 0031972
2017-05-27 20:05 UbitUmarov Note Added: 0031973
2017-05-27 20:07 watcher64 Note Added: 0031974
2017-05-27 20:11 UbitUmarov Note Added: 0031975
2017-05-27 20:23 UbitUmarov Note Added: 0031976
2017-05-27 20:28 UbitUmarov Note Added: 0031977
2017-05-27 20:28 watcher64 Note Added: 0031978
2017-05-27 20:30 UbitUmarov Note Added: 0031979
2017-05-27 21:20 UbitUmarov Note Added: 0031980
2017-05-27 22:22 UbitUmarov Note Added: 0031981
2017-05-27 23:30 UbitUmarov Note Added: 0031982


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker