[Opensim-users] Memory usage
Thomas Ringate
tringate at gmail.com
Fri May 29 00:24:13 UTC 2015
I tested the following 10 versions of opensim built for the OSgrid.
The times are from the monitor program "monit",and are the uptime.
The test process was;
Place a fresh copy of opensim as distributed, copy the needed .ini files
into the simulator,
Restart the region. (This causes "monit" to detect a new PID and resets
it's "uptime" to be zero)
7 minute tick, login with "coolvl" with standard unscripted avatar, record
metrics.
10 minute tick, log off, record metrics.
15 minute tick record metrics.
The 7 minute tick metrics represents the IDLE usage of a freshly started and
un-visited region.
(the reason for waiting 7 minutes was to give ample time to compile the 626
scripts as an initial load)
The 10 minute tick metrics represents the usage after an avatar has logged
in and been on the region for approximately 3 minutes.
The 15 minute tick represents the usage 5 minutes after an avatar has left
the region and no other usage of the region has taken place.
Region used: Sanctuary Prims= 7985 Scripts= 626
------------------------------------------------------------------- idle
1 avatar avatar leave
-------------------------------------------------------- Time mem kb CPU%
Time mem kb CPU% Time mem kb CPU%
osgrid-opensim-01162015.v0.8.1.97ac80d 7m 574952k 3.0
osgrid-opensim-01172015.v0.8.1.1f04e1b 7m 585076k 2.8 27m 513324k 2.9
30m 513364k 2.6
osgrid-opensim-02212015.v0.8.1.7b9ad11 7m 565480k 2.5 10m 560624k 2.9
15m 536468k 2.5
osgrid-opensim-03142015.v0.8.1.8b13e4e 7m 568560k 2.6 10m 576828k 2.9
15m 567436k 2.5
osgrid-opensim-03172015.v0.8.2.83e58eb 7m 580736k 2.4 10m 583992k 2.8
15m 572664k 2.5
osgrid-opensim-04052015.v0.8.2.8d66284 7m 564208k 2.5 10m 574464k 2.8
15m 574704k 2.5
osgrid-opensim-04122015.v0.8.2.d96d31b 7m 574392k 2.6 10m 585156k 3.0
15m 546868k 2.6
osgrid-opensim-05072015.v0.8.2.c74cef0 7m 583924k 2.6 10m 1139092k 2.7
15m 1141972k 2.5
osgrid-opensim-05092015.v0.8.2.adf0f49 7m 578632k 2.5 10m 1207972k 2.9
15m 1208696k 2.5
osgrid-opensim-05232015.v0.8.2.abb3bb6 7m 585384K 2.6 10m 584852k 5.5
15m 1165304k 2.5
Additional comments:
This problem appeared for me in the May 7, 2015 OSgrid release build. It is
present in the latest build which was May 23, 2015.
The initial IDLE memory appears to be nearly the same for all versions
tested.
Prior to the May 7th release memory usage fluctuated very little when
avatars came and went.
After the May 7 release memory usage goes up dramatically, for EACH avatar
that arrives and does not go back down.
There was a change on how memory is used between the May 9 and May 23
builds. The first builds used the memory much faster, and CPU usage would
rise quickly last for a short time and then drop back down.
In the May 23 release, CPU usage goes up much higher, holds at that high
usage level much longer but does drop back to the "idle" level. Memory is
slower to be increased, but still rises to over double that of the idle rate
and does not drop back to the idle level ever.
The value of AppDomainLoading has been set to "false" in the OSgrid builds
for all the builds distributed this year that I have, that is what all of my
tests were run with.
However reading that parameter it reads to me that having it set to FALSE
would cause this exact problem.
That does not explain why it never caused this problem prior to the May 7th
build.
I have "monit" configured to allow both memory and CPU usage to go over a
set limit but if it remains above that limit for longer than what I believe
to be a reasonable time, in my case 3 minutes, it will restart the region.
>From 2011 until May 7th 2015 it was rare any of my regions were restarted by
"monit". After the May 9th release "monit" would restart my regions as soon
as any avatar visited it. I had set the memory limit to be 1.3GB prior to
May 7th. Now setting it at 5GB still results in some regions restarting a
few times every day.
Until the May 9th release none of my four servers had ever used the swap
file. Now the usage of the swap file is in the mid 20%. Lag has become a
problem on my regions, and system CPU usage is much higher than it ever was
prior to the May 7th release.
To suggest nothing is wrong with whatever was done to opensim between the
March release I tested and the May 7th release is very difficult to accept.
If the only problem was higher CPU usage I would accept it, but since
performance and reliability have also been diminished this is not something
that can be ignored.
Memory that gets used and not released is not a good thing. This is
happening across three released of Fedora linux. Fedora 19, 20, and 21, on
four different servers.
I do not know if I should report this in mantis . Usually when I report
something it is already being worked on under some other report using
language I would not think to search on.
Tom
-----Original Message-----
From: Luisillo Contepomi
Sent: Thursday, May 28, 2015 12:00 PM
To: opensim-users at opensimulator.org
Subject: Re: [Opensim-users] Memory usage
I have setup always false in "AppDomainLoading" in all my regions.
Is very important the difference.
Results with "AppDomainLoading" in OpenSim.ini
( 4467 scripts in this region, no avatars login.)
AppDomainLoading = "false"
--------------------------------------------
MEMORY STATISTICS
Heap allocated to OpenSim : 1456 MB
Last heap allocation rate : 20.647 MB/s
Average heap allocation rate: 9.178 MB/s
Process memory : 1868 MB
AppDomainLoading = "true"
----------------------------------------
MEMORY STATISTICS
Heap allocated to OpenSim : 918 MB
Last heap allocation rate : 20.23 MB/s
Average heap allocation rate: 12.022 MB/s
Process memory : 3570 MB
---------------------------------------
Status of XEngine instance for Continente
Scripts loaded : 4467
Scripts waiting for load : 0
Max threads : 100
Min threads : 2
Allocated threads : 100
In use threads : 100
Work items waiting : 104
Events queued : 418
Events processed : 165762
Sensors : 157
Dataserver requests : 0
Timers : 702
Listeners : 78
Regards,
Luisillo
2015-05-28 17:06 GMT+02:00 Thomas Ringate <tringate at gmail.com>:
> Very interesting. My setup is quite different in that I do not run the
> grid
> but rather use the OSgrid. I only run the regions. That could be a huge
> difference. However since OSgrid is officially designated as the
> "opensim"
> test grid, I would hope the developers are serious about the results
> people
> are having who use that grid.
>
> I am in the process of trying to discover exactly when this problem
> started
> for me as was earlier requested.
>
> The different versions I can test with are these releases from OSgrid:
> osgrid-opensim-01162015.v0.8.1.97ac80d
> osgrid-opensim-01172015.v0.8.1.1f04e1b
> osgrid-opensim-02212015.v0.8.1.7b9ad11
> osgrid-opensim-03142015.v0.8.1.8b13e4e
> osgrid-opensim-03172015.v0.8.2.83e58eb
> osgrid-opensim-04052015.v0.8.2.8d66284
> osgrid-opensim-04122015.v0.8.2.d96d31b
> osgrid-opensim-05072015.v0.8.2.c74cef0
> osgrid-opensim-05092015.v0.8.2.adf0f49
> osgrid-opensim-05232015.v0.8.2.abb3bb6
>
> Later today I will cycle all of these releases on one of my regions and
> run
> a test using "monit" with my original parameters set to see which version
> it
> was that started using so much simulator memory. The change control on my
> home network is not near as professional as what I lived with at IBM for
> 31
> years. Now I depend on my own memory. That is not a very good choice in
> my
> case. Luisillo's testing was far more accurate with excellent metrics to
> compare. Bravo.
>
> Tom
>
>
> -----Original Message----- From: Luisillo Contepomi
> Sent: Thursday, May 28, 2015 5:07 AM
> To: opensim-users at opensimulator.org
> Subject: Re: [Opensim-users] Memory usage
>
>
> I have not problems or issues with memory.
> My memory test results:
>
> Grid and sim in the same machine.
> Windows7 - 64 - RAM 8GB
> Intel Core i7-3770 at 3.40GHz
> Users in this test is out in other machines connecting by a 100Mb network.
>
> All users in this test are using Radegast. All have Inventory with
> 1200 Items aprox.
> and worn scripts as AO, radar and resizers in hair and shoes.
>
> Continente is a Var Region 1024x1024
> Prims used 55.048
> Scripts 4.499
>
> Region (Continente) # xengine status
> Status of XEngine instance for Continente
> Scripts loaded : 4499
> Scripts waiting for load : 0
> Max threads : 100
> Min threads : 2
> Allocated threads : 100
> In use threads : 42
> Work items waiting : 0
> Events queued : 636
> Events processed : 1028273
> Sensors : 157
> Dataserver requests : 0
> Timers : 715
> Listeners : 85
>
> From tast manager in K
> CPU 1 from 8 at 70 - 75% all time during test.
> -----------------
> Memory OpenSim.exe / MySql server
> Start region at. 9.39h
> ------------------------------------------------------------
> 1.613.652 no users 9.40h 652.976
> 2.140.920 no users 9.45h "
> ------------------------------------------------------------
> finish scripts load 9.46h "
>
> I order in console "fcache clear file" 9.47h Im not using memory cache. "
> ------------------------------------------------------------
> 2.142.964 no users 9.47h "
> 2.134.240 no users 9.48h 653.140
> 1.468.540 no users 9.49h
> 1.148.276 no users 9.50h (fcache clear finished.)
>
> Test with 2 users
> -------------------------------------------------------------
> 1.148.384 Login in one user 9.54H (RADEGAST)
> 1.185.828 with 1 user 9.56h
> 1.337.280 with 2 users 9.58h 734.548
>
> 1.389.148 2 users one is searching in his inventory 735.900
>
> 1.411.896 logout one user 10.01h 736.452
> 1.515.700 1 user 10.02H
> 1.271.528 1 user 10.03H 736.468
> ---------------------------------------------------
> 1.257.620 logout user No users in 10.04H 736.692
> 1.221.208 no users 10.05h 736.692
> 1.207.296 no users 10.07h 736.692
> 1.204.980 no users 10.07h 735.660
> 1.204.736 no users 10.09h 735.660
>
> ------------simultaneus 8 users login at 10.10h
> ---------------------------------------------------------------
>
> 1.228.172 10.10h.
> 1.427.396 10.11h 5 users login
> 1.699.788 10.12h 8 users login
> 1.937.552 10.13h 8 users in home
> 2.008.496 10.14h 8 users in home 745.468
> 2.010.358 10.15h 8 users in home 745.928
> 1.960.540 10.14h 8 users in home 745.944
> 1.887.942 10.15h 8 users in home "
> 1.946.372 10.16h 8 users in home "
> 1.902.660 10.17h 8 users in home location "
> 1.927.866 10.18h " "
> 1.872.444 10.19h " "
> 1.885.264 10.20h " "
> 1.838.972 10.23h " "
> 1.908.056 10.25h "
>
> Now logout all users one by minute.
>
> 1 - 1.906.736 10.26h 746.528
> 2 - 1.907.524 10.27h 746.532
> 3 - 1.876.456 10.28h 746.612
> 4 - 1.884.836 10.29h "
> 5 - 1.788.868 10.30h 746.852
> 6 - 1.732.212 10.31h "
> 7 - 1.756.340 10.32h "
> 8 - 1.648.408 10.33h "
> NO USERS
> 1.629.464 10.34h "
> 1.618.988 10.36h
> 1.434.444 10.37h
>
> Robust.exe always near 136.570K
> ---------------------------------------------------------
> 10.17h When have 8 users STATS:
> ---------------------------------------------------------
> Region (Continente) # show stats
>
>
> CONNECTION STATISTICS
> Client logouts due to no data receive timeout: 0
>
>
> SAMPLE FRAME STATISTICS
> Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr
> ScrLPS
> 1.00 55 55.6 0.0 8 0 56152 25 4501
> 10991070
>
>
> PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt
> AgntFt
> ImgsFt
> 20 123 0 0 1142 18.1 0.0 0.4 0.0 0.6
> 0.2
>
>
> MEMORY STATISTICS
> Heap allocated to OpenSim : 1647 MB
> Last heap allocation rate : 8.061 MB/s
> Average heap allocation rate: 9.906 MB/s
> Process memory : 2007 MB
>
> 10.46H NO USERS STATS
> ------------------------------------------------------
> Region (Continente) # show stats
>
>
> CONNECTION STATISTICS
> Client logouts due to no data receive timeout: 0
>
>
> SAMPLE FRAME STATISTICS
> Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr
> ScrLPS
> 1.00 55 55.3 0.0 0 0 54875 24 4451
> 13512840
>
>
> PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt
> AgntFt
> ImgsFt
> 13 78 0 0 0 18.3 0.0 0.6 0.0 0.0
> 0.0
>
>
> MEMORY STATISTICS
> Heap allocated to OpenSim : 768 MB
> Last heap allocation rate : 1.732 MB/s
> Average heap allocation rate: 2.493 MB/s
> Process memory : 1493 MB
>
> For me all is ok.
>
>
>
>
> 2015-05-28 1:53 GMT+02:00 Teravus Ovares <teravus at gmail.com>:
>>
>> https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
>> successful at identifying memory issues in the past.
>>
>> That said, it's not free and there may be free alternatives..... but be
>> advised you want one that supports managed memory... and one that
>> works
>> well with Mono.Addins. Many free ones in the past have had issues
>> dynamically loading modules with Mono.addins. If the modules are not
>> loaded, you will see the server come up with no regions and it will be
>> pretty useless for testing. Testing physics is a bit harder because it
>> is
>> both managed and unmanaged... so you'll need to examine the pinned
>> memory.
>> Also, note that while running the simulator in a profiler such as
>> dotMemory,
>> it will run noticeably slower depending on how you do the profiling.
>>
>> Regards
>>
>> Teravus
>>
>> On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <marceled9 at gmail.com>
>> wrote:
>>>
>>>
>>> The memory consumption is troubling.
>>>
>>> It seems to be an cross platform issue, it happens on linux, mac and
>>> also
>>> windows.
>>>
>>> The script engine can be responsible for the skyrocketing of the memory
>>> usage when an avatar enters the region.
>>> The inventory system is also called as a possible suspect.
>>> Also the physics system can be un issuer.
>>>
>>> This throws up a question, how can the memory usage be broken down, to
>>> see
>>> what part is to blame ?
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> Opensim-users at opensimulator.org
>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>
>>
>>
>> _______________________________________________
>> Opensim-users mailing list
>> Opensim-users at opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
_______________________________________________
Opensim-users mailing list
Opensim-users at opensimulator.org
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
More information about the Opensim-users
mailing list