[Opensim-users] Memory usage
Dahlia Trimble
dahliatrimble at gmail.com
Fri May 29 22:25:41 UTC 2015
I'm not sure if this is your problem but some recent changes that improve
inventory loading may be contributing to this. As I'm told (I haven't
verified this personally), prior versions would make many redundant
requests to the inventory server. Now, fewer requests are made and they may
be for larger chunks of data. Typically when many requests are made via
http, some throttling (intentional and not intentional) occurs which would
free the CPU for other tasks, or show it as idle. Now, less throttling may
be necessary as fewer irrelevant requests are made and the ones that
complete can be used effectively which could make the CPU usage appear
higher and require more memory, especially for large inventories. The
nature of the .NET managed runtime environment is such that the memory is
only freed when the garbage collection process is invoked which could be
significantly delayed, which is probably why you see the memory use
decrease over time.
I'm not saying this is definitely your problem, only that it's something
I'm aware of that might be a cause of your symptoms you describe. If so,
it's likely not a bug but rather a difference in how things are working
with the goal of improved performance.
On Fri, May 29, 2015 at 1:35 PM, Thomas Ringate <tringate at gmail.com> wrote:
> Luisillo,
>
> I have opened a new mantis on this issue.
> [opensim 0007593]: High memory usage when avatar arrives and is never
> released
>
> My fingers are crossed that I have not duplicated some other mantis that
> is reporting the same problem.
>
> For my regions the problem is very easy to see. I get the impression that
> is not the case for many others who do not have this problem.
>
> My CPU usage is different than before, but it does return to normal levels
> soon after an avatar leaves. What is different is that it goes much higher
> now, and lasts longer.
>
> I did watch it while three avatars arrived and both CPU and memory went
> nuts. CPU returned to normal but memory remained at a level four times the
> idle level. When they left, memory stayed at the high level. It is now a
> few hours later, and memory is still at the high level. A region restart
> put everything back to normal until the next time it is visited.
>
> Tom
>
>
> -----Original Message----- From: Luisillo Contepomi
> Sent: Friday, May 29, 2015 3:53 AM
>
> To: opensim-users at opensimulator.org
> Subject: Re: [Opensim-users] Memory usage
>
> Im looking about this in mantis.
> Was reported from 0.7.3.X but is closed.
> Please report it again for the new versión.
>
> http://opensimulator.org/mantis/view.php?id=6030
> http://opensimulator.org/mantis/view.php?id=6031
>
> Today I am observing cpu consuption... is hight : from 70% to 100% all
> time for one cpu.
>
>
>
> 2015-05-29 2:24 GMT+02:00 Thomas Ringate <tringate at gmail.com>:
>
>> I tested the following 10 versions of opensim built for the OSgrid.
>>
>> The times are from the monitor program "monit",and are the uptime.
>> The test process was;
>> Place a fresh copy of opensim as distributed, copy the needed .ini files
>> into the simulator,
>> Restart the region. (This causes "monit" to detect a new PID and resets
>> it's "uptime" to be zero)
>> 7 minute tick, login with "coolvl" with standard unscripted avatar, record
>> metrics.
>> 10 minute tick, log off, record metrics.
>> 15 minute tick record metrics.
>> The 7 minute tick metrics represents the IDLE usage of a freshly started
>> and
>> un-visited region.
>> (the reason for waiting 7 minutes was to give ample time to compile the
>> 626
>> scripts as an initial load)
>> The 10 minute tick metrics represents the usage after an avatar has logged
>> in and been on the region for approximately 3 minutes.
>> The 15 minute tick represents the usage 5 minutes after an avatar has left
>> the region and no other usage of the region has taken place.
>>
>> Region used: Sanctuary Prims= 7985 Scripts= 626
>> ------------------------------------------------------------------- idle
>> 1
>> avatar avatar leave
>> -------------------------------------------------------- Time mem kb CPU%
>> Time mem kb CPU% Time mem kb CPU%
>> osgrid-opensim-01162015.v0.8.1.97ac80d 7m 574952k 3.0
>> osgrid-opensim-01172015.v0.8.1.1f04e1b 7m 585076k 2.8 27m 513324k
>> 2.9
>> 30m 513364k 2.6
>> osgrid-opensim-02212015.v0.8.1.7b9ad11 7m 565480k 2.5 10m 560624k
>> 2.9
>> 15m 536468k 2.5
>> osgrid-opensim-03142015.v0.8.1.8b13e4e 7m 568560k 2.6 10m 576828k
>> 2.9
>> 15m 567436k 2.5
>> osgrid-opensim-03172015.v0.8.2.83e58eb 7m 580736k 2.4 10m 583992k
>> 2.8
>> 15m 572664k 2.5
>> osgrid-opensim-04052015.v0.8.2.8d66284 7m 564208k 2.5 10m 574464k
>> 2.8
>> 15m 574704k 2.5
>> osgrid-opensim-04122015.v0.8.2.d96d31b 7m 574392k 2.6 10m 585156k
>> 3.0
>> 15m 546868k 2.6
>> osgrid-opensim-05072015.v0.8.2.c74cef0 7m 583924k 2.6 10m 1139092k
>> 2.7
>> 15m 1141972k 2.5
>> osgrid-opensim-05092015.v0.8.2.adf0f49 7m 578632k 2.5 10m 1207972k
>> 2.9
>> 15m 1208696k 2.5
>> osgrid-opensim-05232015.v0.8.2.abb3bb6 7m 585384K 2.6 10m 584852k
>> 5.5
>> 15m 1165304k 2.5
>>
>> Additional comments:
>>
>> This problem appeared for me in the May 7, 2015 OSgrid release build. It
>> is
>> present in the latest build which was May 23, 2015.
>> The initial IDLE memory appears to be nearly the same for all versions
>> tested.
>> Prior to the May 7th release memory usage fluctuated very little when
>> avatars came and went.
>> After the May 7 release memory usage goes up dramatically, for EACH avatar
>> that arrives and does not go back down.
>> There was a change on how memory is used between the May 9 and May 23
>> builds. The first builds used the memory much faster, and CPU usage would
>> rise quickly last for a short time and then drop back down.
>> In the May 23 release, CPU usage goes up much higher, holds at that high
>> usage level much longer but does drop back to the "idle" level. Memory is
>> slower to be increased, but still rises to over double that of the idle
>> rate
>> and does not drop back to the idle level ever.
>> The value of AppDomainLoading has been set to "false" in the OSgrid builds
>> for all the builds distributed this year that I have, that is what all of
>> my
>> tests were run with.
>> However reading that parameter it reads to me that having it set to FALSE
>> would cause this exact problem.
>> That does not explain why it never caused this problem prior to the May
>> 7th
>> build.
>>
>> I have "monit" configured to allow both memory and CPU usage to go over a
>> set limit but if it remains above that limit for longer than what I
>> believe
>> to be a reasonable time, in my case 3 minutes, it will restart the region.
>>
>> From 2011 until May 7th 2015 it was rare any of my regions were restarted
>> by
>> "monit". After the May 9th release "monit" would restart my regions as
>> soon
>> as any avatar visited it. I had set the memory limit to be 1.3GB prior to
>> May 7th. Now setting it at 5GB still results in some regions restarting a
>> few times every day.
>>
>> Until the May 9th release none of my four servers had ever used the swap
>> file. Now the usage of the swap file is in the mid 20%. Lag has become a
>> problem on my regions, and system CPU usage is much higher than it ever
>> was
>> prior to the May 7th release.
>>
>> To suggest nothing is wrong with whatever was done to opensim between the
>> March release I tested and the May 7th release is very difficult to
>> accept.
>>
>> If the only problem was higher CPU usage I would accept it, but since
>> performance and reliability have also been diminished this is not
>> something
>> that can be ignored.
>>
>> Memory that gets used and not released is not a good thing. This is
>> happening across three released of Fedora linux. Fedora 19, 20, and 21,
>> on
>> four different servers.
>>
>> I do not know if I should report this in mantis . Usually when I report
>> something it is already being worked on under some other report using
>> language I would not think to search on.
>>
>> Tom
>>
>>
>>
>> -----Original Message----- From: Luisillo Contepomi
>> Sent: Thursday, May 28, 2015 12:00 PM
>> To: opensim-users at opensimulator.org
>> Subject: Re: [Opensim-users] Memory usage
>>
>> I have setup always false in "AppDomainLoading" in all my regions.
>> Is very important the difference.
>>
>>
>> Results with "AppDomainLoading" in OpenSim.ini
>> ( 4467 scripts in this region, no avatars login.)
>>
>> AppDomainLoading = "false"
>> --------------------------------------------
>> MEMORY STATISTICS
>> Heap allocated to OpenSim : 1456 MB
>> Last heap allocation rate : 20.647 MB/s
>> Average heap allocation rate: 9.178 MB/s
>> Process memory : 1868 MB
>>
>> AppDomainLoading = "true"
>> ----------------------------------------
>> MEMORY STATISTICS
>> Heap allocated to OpenSim : 918 MB
>> Last heap allocation rate : 20.23 MB/s
>> Average heap allocation rate: 12.022 MB/s
>> Process memory : 3570 MB
>>
>> ---------------------------------------
>> Status of XEngine instance for Continente
>> Scripts loaded : 4467
>> Scripts waiting for load : 0
>> Max threads : 100
>> Min threads : 2
>> Allocated threads : 100
>> In use threads : 100
>> Work items waiting : 104
>> Events queued : 418
>> Events processed : 165762
>> Sensors : 157
>> Dataserver requests : 0
>> Timers : 702
>> Listeners : 78
>>
>> Regards,
>> Luisillo
>>
>> 2015-05-28 17:06 GMT+02:00 Thomas Ringate <tringate at gmail.com>:
>>
>>>
>>> Very interesting. My setup is quite different in that I do not run the
>>> grid
>>> but rather use the OSgrid. I only run the regions. That could be a huge
>>> difference. However since OSgrid is officially designated as the
>>> "opensim"
>>> test grid, I would hope the developers are serious about the results
>>> people
>>> are having who use that grid.
>>>
>>> I am in the process of trying to discover exactly when this problem
>>> started
>>> for me as was earlier requested.
>>>
>>> The different versions I can test with are these releases from OSgrid:
>>> osgrid-opensim-01162015.v0.8.1.97ac80d
>>> osgrid-opensim-01172015.v0.8.1.1f04e1b
>>> osgrid-opensim-02212015.v0.8.1.7b9ad11
>>> osgrid-opensim-03142015.v0.8.1.8b13e4e
>>> osgrid-opensim-03172015.v0.8.2.83e58eb
>>> osgrid-opensim-04052015.v0.8.2.8d66284
>>> osgrid-opensim-04122015.v0.8.2.d96d31b
>>> osgrid-opensim-05072015.v0.8.2.c74cef0
>>> osgrid-opensim-05092015.v0.8.2.adf0f49
>>> osgrid-opensim-05232015.v0.8.2.abb3bb6
>>>
>>> Later today I will cycle all of these releases on one of my regions and
>>> run
>>> a test using "monit" with my original parameters set to see which version
>>> it
>>> was that started using so much simulator memory. The change control on
>>> my
>>> home network is not near as professional as what I lived with at IBM for
>>> 31
>>> years. Now I depend on my own memory. That is not a very good choice in
>>> my
>>> case. Luisillo's testing was far more accurate with excellent metrics to
>>> compare. Bravo.
>>>
>>> Tom
>>>
>>>
>>> -----Original Message----- From: Luisillo Contepomi
>>> Sent: Thursday, May 28, 2015 5:07 AM
>>> To: opensim-users at opensimulator.org
>>> Subject: Re: [Opensim-users] Memory usage
>>>
>>>
>>> I have not problems or issues with memory.
>>> My memory test results:
>>>
>>> Grid and sim in the same machine.
>>> Windows7 - 64 - RAM 8GB
>>> Intel Core i7-3770 at 3.40GHz
>>> Users in this test is out in other machines connecting by a 100Mb
>>> network.
>>>
>>> All users in this test are using Radegast. All have Inventory with
>>> 1200 Items aprox.
>>> and worn scripts as AO, radar and resizers in hair and shoes.
>>>
>>> Continente is a Var Region 1024x1024
>>> Prims used 55.048
>>> Scripts 4.499
>>>
>>> Region (Continente) # xengine status
>>> Status of XEngine instance for Continente
>>> Scripts loaded : 4499
>>> Scripts waiting for load : 0
>>> Max threads : 100
>>> Min threads : 2
>>> Allocated threads : 100
>>> In use threads : 42
>>> Work items waiting : 0
>>> Events queued : 636
>>> Events processed : 1028273
>>> Sensors : 157
>>> Dataserver requests : 0
>>> Timers : 715
>>> Listeners : 85
>>>
>>> From tast manager in K
>>> CPU 1 from 8 at 70 - 75% all time during test.
>>> -----------------
>>> Memory OpenSim.exe / MySql server
>>> Start region at. 9.39h
>>> ------------------------------------------------------------
>>> 1.613.652 no users 9.40h 652.976
>>> 2.140.920 no users 9.45h "
>>> ------------------------------------------------------------
>>> finish scripts load 9.46h "
>>>
>>> I order in console "fcache clear file" 9.47h Im not using memory cache. "
>>> ------------------------------------------------------------
>>> 2.142.964 no users 9.47h "
>>> 2.134.240 no users 9.48h 653.140
>>> 1.468.540 no users 9.49h
>>> 1.148.276 no users 9.50h (fcache clear finished.)
>>>
>>> Test with 2 users
>>> -------------------------------------------------------------
>>> 1.148.384 Login in one user 9.54H (RADEGAST)
>>> 1.185.828 with 1 user 9.56h
>>> 1.337.280 with 2 users 9.58h 734.548
>>>
>>> 1.389.148 2 users one is searching in his inventory 735.900
>>>
>>> 1.411.896 logout one user 10.01h 736.452
>>> 1.515.700 1 user 10.02H
>>> 1.271.528 1 user 10.03H 736.468
>>> ---------------------------------------------------
>>> 1.257.620 logout user No users in 10.04H 736.692
>>> 1.221.208 no users 10.05h 736.692
>>> 1.207.296 no users 10.07h 736.692
>>> 1.204.980 no users 10.07h 735.660
>>> 1.204.736 no users 10.09h 735.660
>>>
>>> ------------simultaneus 8 users login at 10.10h
>>> ---------------------------------------------------------------
>>>
>>> 1.228.172 10.10h.
>>> 1.427.396 10.11h 5 users login
>>> 1.699.788 10.12h 8 users login
>>> 1.937.552 10.13h 8 users in home
>>> 2.008.496 10.14h 8 users in home 745.468
>>> 2.010.358 10.15h 8 users in home 745.928
>>> 1.960.540 10.14h 8 users in home 745.944
>>> 1.887.942 10.15h 8 users in home "
>>> 1.946.372 10.16h 8 users in home "
>>> 1.902.660 10.17h 8 users in home location "
>>> 1.927.866 10.18h " "
>>> 1.872.444 10.19h " "
>>> 1.885.264 10.20h " "
>>> 1.838.972 10.23h " "
>>> 1.908.056 10.25h "
>>>
>>> Now logout all users one by minute.
>>>
>>> 1 - 1.906.736 10.26h 746.528
>>> 2 - 1.907.524 10.27h 746.532
>>> 3 - 1.876.456 10.28h 746.612
>>> 4 - 1.884.836 10.29h "
>>> 5 - 1.788.868 10.30h 746.852
>>> 6 - 1.732.212 10.31h "
>>> 7 - 1.756.340 10.32h "
>>> 8 - 1.648.408 10.33h "
>>> NO USERS
>>> 1.629.464 10.34h "
>>> 1.618.988 10.36h
>>> 1.434.444 10.37h
>>>
>>> Robust.exe always near 136.570K
>>> ---------------------------------------------------------
>>> 10.17h When have 8 users STATS:
>>> ---------------------------------------------------------
>>> Region (Continente) # show stats
>>>
>>>
>>> CONNECTION STATISTICS
>>> Client logouts due to no data receive timeout: 0
>>>
>>>
>>> SAMPLE FRAME STATISTICS
>>> Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr
>>> ScrLPS
>>> 1.00 55 55.6 0.0 8 0 56152 25 4501
>>> 10991070
>>>
>>>
>>> PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt
>>> AgntFt
>>> ImgsFt
>>> 20 123 0 0 1142 18.1 0.0 0.4 0.0 0.6
>>> 0.2
>>>
>>>
>>> MEMORY STATISTICS
>>> Heap allocated to OpenSim : 1647 MB
>>> Last heap allocation rate : 8.061 MB/s
>>> Average heap allocation rate: 9.906 MB/s
>>> Process memory : 2007 MB
>>>
>>> 10.46H NO USERS STATS
>>> ------------------------------------------------------
>>> Region (Continente) # show stats
>>>
>>>
>>> CONNECTION STATISTICS
>>> Client logouts due to no data receive timeout: 0
>>>
>>>
>>> SAMPLE FRAME STATISTICS
>>> Dilatn SimFPS PhyFPS AgntUp RootAg ChldAg Prims AtvPrm AtvScr
>>> ScrLPS
>>> 1.00 55 55.3 0.0 0 0 54875 24 4451
>>> 13512840
>>>
>>>
>>> PktsIn PktOut PendDl PendUl UnackB TotlFt NetFt PhysFt OthrFt
>>> AgntFt
>>> ImgsFt
>>> 13 78 0 0 0 18.3 0.0 0.6 0.0 0.0
>>> 0.0
>>>
>>>
>>> MEMORY STATISTICS
>>> Heap allocated to OpenSim : 768 MB
>>> Last heap allocation rate : 1.732 MB/s
>>> Average heap allocation rate: 2.493 MB/s
>>> Process memory : 1493 MB
>>>
>>> For me all is ok.
>>>
>>>
>>>
>>>
>>> 2015-05-28 1:53 GMT+02:00 Teravus Ovares <teravus at gmail.com>:
>>>
>>>>
>>>>
>>>> https://www.jetbrains.com/dotmemory/ dotmemory has been relatively
>>>> successful at identifying memory issues in the past.
>>>>
>>>> That said, it's not free and there may be free alternatives..... but be
>>>> advised you want one that supports managed memory... and one that
>>>> works
>>>> well with Mono.Addins. Many free ones in the past have had issues
>>>> dynamically loading modules with Mono.addins. If the modules are not
>>>> loaded, you will see the server come up with no regions and it will be
>>>> pretty useless for testing. Testing physics is a bit harder because it
>>>> is
>>>> both managed and unmanaged... so you'll need to examine the pinned
>>>> memory.
>>>> Also, note that while running the simulator in a profiler such as
>>>> dotMemory,
>>>> it will run noticeably slower depending on how you do the profiling.
>>>>
>>>> Regards
>>>>
>>>> Teravus
>>>>
>>>> On Wed, May 27, 2015 at 3:01 AM, M.E. Verhagen <marceled9 at gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>> The memory consumption is troubling.
>>>>>
>>>>> It seems to be an cross platform issue, it happens on linux, mac and
>>>>> also
>>>>> windows.
>>>>>
>>>>> The script engine can be responsible for the skyrocketing of the memory
>>>>> usage when an avatar enters the region.
>>>>> The inventory system is also called as a possible suspect.
>>>>> Also the physics system can be un issuer.
>>>>>
>>>>> This throws up a question, how can the memory usage be broken down, to
>>>>> see
>>>>> what part is to blame ?
>>>>>
>>>>> _______________________________________________
>>>>> Opensim-users mailing list
>>>>> Opensim-users at opensimulator.org
>>>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Opensim-users mailing list
>>>> Opensim-users at opensimulator.org
>>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>>
>>>> _______________________________________________
>>> Opensim-users mailing list
>>> Opensim-users at opensimulator.org
>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>> _______________________________________________
>>> Opensim-users mailing list
>>> Opensim-users at opensimulator.org
>>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>>
>>
>> _______________________________________________
>> Opensim-users mailing list
>> Opensim-users at opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>> _______________________________________________
>> Opensim-users mailing list
>> Opensim-users at opensimulator.org
>> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>>
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at opensimulator.org
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://opensimulator.org/pipermail/opensim-users/attachments/20150529/a745c3bb/attachment-0001.html>
More information about the Opensim-users
mailing list