[Opensim-users] Memory usage
Luisillo Contepomi
luisillocontepomi at gmail.com
Fri May 29 07:53:28 UTC 2015
Im looking about this in mantis.
Was reported from 0.7.3.X but is closed.
Please report it again for the new versión.
http://opensimulator.org/mantis/view.php?id=6030
http://opensimulator.org/mantis/view.php?id=6031
Today I am observing cpu consuption... is hight : from 70% to 100% all
time for one cpu.
2015-05-29 2:24 GMT+02:00 Thomas Ringate <tringate at gmail.com>:
> I tested the following 10 versions of opensim built for the OSgrid.
>
> The times are from the monitor program "monit",and are the uptime.
> The test process was;
> Place a fresh copy of opensim as distributed, copy the needed .ini files
> into the simulator,
> Restart the region. (This causes "monit" to detect a new PID and resets
> it's "uptime" to be zero)
> 7 minute tick, login with "coolvl" with standard unscripted avatar, record
> metrics.
> 10 minute tick, log off, record metrics.
> 15 minute tick record metrics.
> The 7 minute tick metrics represents the IDLE usage of a freshly started and
> un-visited region.
> (the reason for waiting 7 minutes was to give ample time to compile the 626
> scripts as an initial load)
> The 10 minute tick metrics represents the usage after an avatar has logged
> in and been on the region for approximately 3 minutes.
> The 15 minute tick represents the usage 5 minutes after an avatar has left
> the region and no other usage of the region has taken place.
>
> Region used: Sanctuary Prims= 7985 Scripts= 626
> ------------------------------------------------------------------- idle 1
> avatar avatar leave
> -------------------------------------------------------- Time mem kb CPU%
> Time mem kb CPU% Time mem kb CPU%
> osgrid-opensim-01162015.v0.8.1.97ac80d 7m 574952k 3.0
> osgrid-opensim-01172015.v0.8.1.1f04e1b 7m 585076k 2.8 27m 513324k 2.9
> 30m 513364k 2.6
> osgrid-opensim-02212015.v0.8.1.7b9ad11 7m 565480k 2.5 10m 560624k 2.9
> 15m 536468k 2.5
> osgrid-opensim-03142015.v0.8.1.8b13e4e 7m 568560k 2.6 10m 576828k 2.9
> 15m 567436k 2.5
> osgrid-opensim-03172015.v0.8.2.83e58eb 7m 580736k 2.4 10m 583992k 2.8
> 15m 572664k 2.5
> osgrid-opensim-04052015.v0.8.2.8d66284 7m 564208k 2.5 10m 574464k 2.8
> 15m 574704k 2.5
> osgrid-opensim-04122015.v0.8.2.d96d31b 7m 574392k 2.6 10m 585156k 3.0
> 15m 546868k 2.6
> osgrid-opensim-05072015.v0.8.2.c74cef0 7m 583924k 2.6 10m 1139092k 2.7
> 15m 1141972k 2.5
> osgrid-opensim-05092015.v0.8.2.adf0f49 7m 578632k 2.5 10m 1207972k 2.9
> 15m 1208696k 2.5
> osgrid-opensim-05232015.v0.8.2.abb3bb6 7m 585384K 2.6 10m 584852k 5.5
> 15m 1165304k 2.5
>
> Additional comments:
>
> This problem appeared for me in the May 7, 2015 OSgrid release build. It is
> present in the latest build which was May 23, 2015.
> The initial IDLE memory appears to be nearly the same for all versions
> tested.
> Prior to the May 7th release memory usage fluctuated very little when
> avatars came and went.
> After the May 7 release memory usage goes up dramatically, for EACH avatar
> that arrives and does not go back down.
> There was a change on how memory is used between the May 9 and May 23
> builds. The first builds used the memory much faster, and CPU usage would
> rise quickly last for a short time and then drop back down.
> In the May 23 release, CPU usage goes up much higher, holds at that high
> usage level much longer but does drop back to the "idle" level. Memory is
> slower to be increased, but still rises to over double that of the idle rate
> and does not drop back to the idle level ever.
> The value of AppDomainLoading has been set to "false" in the OSgrid builds
> for all the builds distributed this year that I have, that is what all of my
> tests were run with.
> However reading that parameter it reads to me that having it set to FALSE
> would cause this exact problem.
> That does not explain why it never caused this problem prior to the May 7th
> build.
>
> I have "monit" configured to allow both memory and CPU usage to go over a
> set limit but if it remains above that limit for longer than what I believe
> to be a reasonable time, in my case 3 minutes, it will restart the region.
>
> From 2011 until May 7th 2015 it was rare any of my regions were restarted by
> "monit". After the May 9th release "monit" would restart my regions as soon
> as any avatar visited it. I had set the memory limit to be 1.3GB prior to
> May 7th. Now setting it at 5GB still results in some regions restarting a
> few times every day.
>
> Until the May 9th release none of my four servers had ever used the swap
> file. Now the usage of the swap file is in the mid 20%. Lag has become a
> problem on my regions, and system CPU usage is much higher than it ever was
> prior to the May 7th release.
>
> To suggest nothing is wrong with whatever was done to opensim between the
> March release I tested and the May 7th release is very difficult to accept.
>
> If the only problem was higher CPU usage I would accept it, but since
> performance and reliability have also been diminished this is not something
> that can be ignored.
>
> Memory that gets used and not released is not a good thing. This is
> happening across three released of Fedora linux. Fedora 19, 20, and 21, on
> four different servers.
>
> I do not know if I should report this in mantis . Usually when I report
> something it is already being worked on under some other report using
> language I would not think to search on.
>
> Tom
>
>
>
> -----Original Message----- From: Luisillo Contepomi
> Sent: Thursday, May 28, 2015 12:00 PM
> To: opensim-users at opensimulator.org
> Subject: Re: [Opensim-users] Memory usage
>
> I have setup always false in "AppDomainLoading" in all my regions.
> Is very important the difference.
>
>
> Results with "AppDomainLoading" in OpenSim.ini
> ( 4467 scripts in this region, no avatars login.)
>
> AppDomainLoading = "false"
> --------------------------------------------
> MEMORY STATISTICS
> Heap allocated to OpenSim : 1456 MB
> Last heap allocation rate : 20.647 MB/s
> Average heap allocation rate: 9.178 MB/s
> Process memory : 1868 MB
>
> AppDomainLoading = "true"
> ----------------------------------------
> MEMORY STATISTICS
> Heap allocated to OpenSim : 918 MB
> Last heap allocation rate : 20.23 MB/s
> Average heap allocation rate: 12.022 MB/s
> Process memory : 3570 MB
>
> ---------------------------------------
> Status of XEngine instance for Continente
> Scripts loaded : 4467
> Scripts waiting for load : 0
> Max threads : 100
> Min threads : 2
> Allocated threads : 100
> In use threads : 100
> Work items waiting : 104
> Events queued : 418
> Events processed : 165762
> Sensors : 157
> Dataserver requests : 0
> Timers : 702
> Listeners : 78
>
> Regards,
> Luisillo
>
> 2015-05-28 17:06 GMT+02:00 Thomas Ringate <tringate at gmail.com>:
>>
>> Very interesting. My setup is quite different in that I do not run the
>> grid
>> but rather use the OSgrid. I only run the regions. That could be a huge
>> difference. However since OSgrid is officially designated as the
>> "opensim"
>> test grid, I would hope the developers are serious about the results
>> people
>> are having who use that grid.
>>
>> I am in the process of trying to discover exactly when this problem
>> started
>> for me as was earlier requested.
>>
>> The different versions I can test with are these releases from OSgrid:
>> osgrid-opensim-01162015.v0.8.1.97ac80d
>> osgrid-opensim-01172015.v0.8.1.1f04e1b
>> osgrid-opensim-02212015.v0.8.1.7b9ad11
>> osgrid-opensim-03142015.v0.8.1.8b13e4e
>> osgrid-opensim-03172015.v0.8.2.83e58eb
>> osgrid-opensim-04052015.v0.8.2.8d66284
>> osgrid-opensim-04122015.v0.8.2.d96d31b
>> osgrid-opensim-05072015.v0.8.2.c74cef0
>> osgrid-opensim-05092015.v0.8.2.adf0f49
>> osgrid-opensim-05232015.v0.8.2.abb3bb6
>>
>> Later today I will cycle all of these releases on one of my regions and
>> run
>> a test using "monit" with my original parameters set to see which version
>> it
>> was that started using so much simulator memory. The change control on my
>> home network is not near as professional as what I lived with at IBM for
>> 31
>> years. Now I depend on my own memory. That is not a very good choice in
>> my
>> case. Luisillo's testing was far more accurate with excellent metrics to
>> compare. Bravo.
>>
>> Tom
>>
>>
>> -----Original Message----- From: Luisillo Contepomi
>> Sent: Thursday, May 28, 2015 5:07 AM
>> To: opensim-users at opensimulator.org
>> Subject: Re: [Opensim-users] Memory usage
>>
>>
>> I have not problems or issues with memory.
>> My memory test results:
>>
>> Grid and sim in the same machine.
>> Windows7 - 64 - RAM 8GB
>> Intel Core i7-3770 at 3.40GHz
>> Users in this test is out in other machines connecting by a 100Mb network.
>>
>> All users in this test are using Radegast. All have Inventory with
>> 1200 Items aprox.
>> and worn scripts as AO, radar and resizers in hair and shoes.
>>
>> Continente is a Var Region 1024x1024
>> Prims used 55.048
>> Scripts 4.499
>>
>> Region (Continente) # xengine status
>> Status of XEngine instance for Continente
>> Scripts loaded : 4499
>> Scripts waiting for load : 0
>> Max threads : 100
>> Min threads : 2
>> Allocated threads : 100
>> In use threads : 42
>> Work items waiting : 0
>> Events queued : 636
>> Events processed : 1028273
>> Sensors : 157
>> Dataserver requests : 0
>> Timers : 715
>> Listeners : 85
>>
>> From tast manager in K
>> CPU 1 from 8 at 70 - 75% all time during test.
>> -----------------
>> Memory OpenSim.exe / MySql server
>> Start region at. 9.39h
>> ------------------------------------------------------------
>> 1.613.652 no users 9.40h 652.976
>> 2.140.920 no users 9.45h "
>> ------------------------------------------------------------
>> finish scripts load 9.46h "
>>
>> I order in console "fcache clear file" 9.47h Im not using memory cache. "
>> ------------------------------------------------------------
>> 2.142.964 no users 9.47h "
>> 2.134.240 no users 9.48h 653.140
>> 1.468.540 no users 9.49h
>> 1.148.276 no users 9.50h (fcache clear finished.)
>>
>> Test with 2 users
>> -------------------------------------------------------------
>> 1.148.384 Login in one user 9.54H (RADEGAST)
>> 1.185.828 with 1 user 9.56h
>> 1.337.280 with 2 users 9.58h 734.548
>>
>> 1.389.148 2 users one is searching in his inventory 735.900
>>
>> 1.411.896 logout one user 10.01h 736.452
>> 1.515.700 1 user 10.02H
>> 1.271.528 1 user 10.03H 736.468
>> ---------------------------------------------------
>> 1.257.620 logout user No users in 10.04H 736.692
>> 1.221.208 no users 10.05h 736.692
>> 1.207.296 no users 10.07h 736.692
>> 1.204.980 no users 10.07h 735.660
>> 1.204.736 no users 10.09h 735.660
>>
>> ------------simultaneus 8 users login at 10.10h
>> ---------------------------------------------------------------
>>
>> 1.228.172 10.10h.
>> 1.427.396 10.11h 5 users login
>> 1.699.788 10.12h 8 users login
>> 1.937.552 10.13h 8 users in home
>> 2.008.496 10.14h 8 users in home 745.468
>> 2.010.358 10.15h 8 users in home 745.928
>> 1.960.540 10.14h 8 users in home 745.944
>> 1.887.942 10.15h 8 users in home "
>> 1.946.372 10.16h 8 users in home "
>> 1.902.660 10.17h 8 users in home location "
>> 1.927.866 10.18h " "
>> 1.872.444 10.19h " "
>> 1.885.264 10.20h " "
>> 1.838.972 10.23h " "
>> 1.908.056 10.25h "
>>
>> Now logout all users one by minute.
>>
>> 1 - 1.906.736 10.26h 746.528
>> 2 - 1.907.524 10.27h 746.532
>> 3 - 1.876.456 10.28h 746.612
>> 4 - 1.884.836 10.29h "
>> 5 - 1.788.868 10.30h 746.852
>> 6 - 1.732.212 10.31h "
>> 7 - 1.756.340 10.32h "
>> 8 - 1.648.408 10.33h "
>> NO USERS
>> 1.629.464 10.34h "
>> 1.618.988 10.36h
>> 1.434.444 10.37h
>>
>> Robust.exe always near 136.570K
>> ---------------------------------------------------------
>> 10.17h When have 8 users STATS:
>> ---------------------------------------------------------
>> Region (Continente) # show stats
>>
>>
>> CONNECTION STATISTICS
>> Client logouts due to no data receive timeout: 0
>>
>>
>> SAMPLE FRAME STATISTICS
>> Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr
>> ScrLPS
>> 1.00 55 55.6 0.0 8 0 56152 25 4501
>> 10991070
>>
>>
>> PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt
>> AgntFt
>> ImgsFt
>> 20 123 0 0 1142 18.1 0.0 0.4 0.0 0.6
>> 0.2
>>
>>
>> MEMORY STATISTICS
>> Heap allocated to OpenSim : 1647 MB
>> Last heap allocation rate : 8.061 MB/s
>> Average heap allocation rate: 9.906 MB/s
>> Process memory : 2007 MB
>>
>> 10.46H NO USERS STATS
>> ------------------------------------------------------
>> Region (Continente) # show stats
>>
>>
>> CONNECTION STATISTICS
>> Client logouts due to no data receive timeout: 0
>>
>>
>> SAMPLE FRAME STATISTICS
>> Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr
>> ScrLPS
>> 1.00 55 55.3 0.0 0 0 54875 24 4451
>> 13512840
>>
>>
>> PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt
>> AgntFt
>> ImgsFt
>> 13 78 0 0 0 18.3 0.0 0.6 0.0 0.0
>> 0.0
>>
>>
>> MEMORY STATISTICS
>> Heap allocated to OpenSim : 768 MB
>> Last heap allocation rate : 1.732 MB/s
>> Average heap allocation rate: 2.493 MB/s
>> Process memory : 1493 MB
>>
>> For me all is ok.
>>
>>
>>
>>
>> 2015-05-28 1:53 GMT+02:00 Teravus Ovares <teravus at gmail.com>:
>>>
>>>
>>> https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
>>> successful at identifying memory issues in the past.
>>>
>>> That said, it's not free and there may be free alternatives..... but be
>>> advised you want one that supports managed memory... and one that
>>> works
>>> well with Mono.Addins. Many free ones in the past have had issues
>>> dynamically loading modules with Mono.addins. If the modules are not
>>> loaded, you will see the server come up with no regions and it will be
>>> pretty useless for testing. Testing physics is a bit harder because it
>>> is
>>> both managed and unmanaged... so you'll need to examine the pinned
>>> memory.
>>> Also, note that while running the simulator in a profiler such as
>>> dotMemory,
>>> it will run noticeably slower depending on how you do the profiling.
>>>
>>> Regards
>>>
>>> Teravus
>>>
>>> On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <marceled9 at gmail.com>
>>> wrote:
>>>>
>>>>
>>>>
>>>> The memory consumption is troubling.
>>>>
>>>> It seems to be an cross platform issue, it happens on linux, mac and
>>>> also
>>>> windows.
>>>>
>>>> The script engine can be responsible for the skyrocketing of the memory
>>>> usage when an avatar enters the region.
>>>> The inventory system is also called as a possible suspect.
>>>> Also the physics system can be un issuer.
>>>>
>>>> This throws up a question, how can the memory usage be broken down, to
>>>> see
>>>> what part is to blame ?
>>>>
>>>> _______________________________________________
>>>> Opensim-users mailing list
>>>> Opensim-users at opensimulator.org
>>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> Opensim-users at opensimulator.org
>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>
>> _______________________________________________
>> Opensim-users mailing list
>> Opensim-users at opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>> _______________________________________________
>> Opensim-users mailing list
>> Opensim-users at opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
More information about the Opensim-users
mailing list