|Anonymous | Login | Signup for a new account||2020-02-27 21:56 PST|
|Main | My View | View Issues | Change Log | Roadmap | Summary | My Account|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0008630||opensim||[REGION] Scripting Engine||public||2019-11-21 04:35||2020-01-11 04:11|
|Target Version||Fixed in Version|
|Summary||0008630: Memory Leak on YEngine|
|Description||I believe this to be caused by dynamic textures not clearing as well, to a smaller part, http requests data sticking around.|
Been monitoring a small test region with just a few scripts on them, over a day it has accumulated over 3gb of memory, initially started with 200mb:
On other regions with more dynamic textures and http requests over time the leak is much more pronounced so much so I found a simulator consuming 32gb of memory up top its normal 4gb.
|Steps To Reproduce||Spin up a simulator under YEngine|
Create a script with periodic dynamic textures drawing and http requests
Watch memory usage go up and up and up
|Additional Information||I first noticed this after switching from XEngine to YEngine without doing any other changes to the simulator and region, got a monitoring alert because the whole server was out of memory.|
There was an old bug with XEngine not clearing dynamic textures very well, that has since been resolved though.
|Tags||No tags attached.|
|Git Revision or version number|
|Run Mode||Grid (1 Region per Sim) , Grid (Multiple Regions per Sim)|
|Environment||Mono / Linux64|
edited on: 2019-11-21 05:01
dynamic texture code is the same on X and Y. Engines only see UUIDs and not the bulk data, so one can leak if the other doesnt
As tampa and i seen, one of is test scripts shown no leaks on mono 6.0.??
He is using last stable 6.4 also with a new libgdi...
I will test with mono 6.0 as well after reverting back to snapshot and building new binaries. If that fixes it then we know it is probably related to changes in 6.4 release which is current stable for mono. This however means regardless it needs looking at because naturally mono website redirects to latest stable release so people will install that.
Assuming either changes to gc or libgdiplus are the culprits, but as Ubit said figuring that out might be very hard :S
Will report back with what I find on mono 6.0, for now I guess a careful warning not to use 6.4 in the meantime(add to wiki perhaps?)
So the issue can be traced to libgdiplus. After installing the distro version from ubuntu mirrors the problem went away.
apt purge libgdiplus
apt install libgdiplus=4.2-1ubuntu1
This means whatever changed in libgdiplus 6.0.4 is the cause of the leak under YEngine, meaning then either sticking to old version of libgdiplus or fixing what's broken :/
I created 4 cubes with the scripts tampa posted in a link on #opensim-dev@freenode yesterday on a simulator running XEngine and a simulator running YEngine.
In a first run over night (roughly 9 hours) I started only one of the four scripts on the simulator with XEngine, but I've seen a systemwide increase of 7 GB.
Next I started a second run where I started all four scripts on both engines and monitored each simulator separately. After 6 hours it summed up to an identical increase of roughly 2.5 GB on each simulator.
I tested on Ubuntu 18.04 with Mono 220.127.116.11 with libgdiplus 6.0.4.
P.S. Currently there is an open pull request related to resource leaks on https://github.com/mono/libgdiplus. [^] Not sure this will resolve the issue in the furture.
|It looks ok after testing 14 hours on Ubuntu 18.04 running Mono 18.104.22.168 with libgdiplus 4.2-2. The problem definitely seems to be in libgdiplus.|
|Confirming this for now, though whether this warrants changes or just some time to wait for updates on libgdiplus is in the stars|
Apparently there were some updates to libgdiplus recently that may help resolve this issue, however no new release has been made yet so these changes only apply if built from their master. Since it then also depends on mono to update to that latest version, seeing as distros are unlikely to do so, it may be even longer until an update resolves this.
I propose we resolve this temporarily by shipping our own libgdiplus along with either bin or libomv and only load a different version if the version on the system is newer than our own, that way we can either ship an older working version or the latest from their master to hopefully solve this before every other simulator starts choking on textures.
Else I feel the only way is a big red warning on the wiki regarding the install of mono requiring the use of older versions of libgdiplus :/
|2019-11-21 04:35||tampa||New Issue|
|2019-11-21 05:01||UbitUmarov||Note Added: 0035898|
|2019-11-21 05:01||UbitUmarov||Note Edited: 0035898||View Revisions|
|2019-11-21 06:10||UbitUmarov||Note Added: 0035899|
|2019-11-21 06:14||tampa||Note Added: 0035900|
|2019-11-21 06:52||tampa||Note Added: 0035901|
|2019-11-21 08:37||piusnoel||Note Added: 0035902|
|2019-11-22 02:49||piusnoel||Note Added: 0035903|
|2019-11-22 05:23||tampa||Note Added: 0035904|
|2019-11-22 05:23||tampa||Status||new => confirmed|
|2019-11-28 05:39||tampa||Note Added: 0035915|
|2019-11-29 22:32||geektech12||Note Added: 0035916|
|2019-11-29 22:34||geektech12||Note Edited: 0035916||View Revisions|
|2019-11-29 22:34||UbitUmarov||Note Deleted: 0035916|
|2019-11-29 22:37||geektech12||Note Added: 0035917|
|2019-11-29 22:37||geektech12||Note Edited: 0035917||View Revisions|
|2019-11-29 23:43||UbitUmarov||Note Deleted: 0035917|
|2020-01-11 04:11||aol_gold_support||Note Added: 0036040|
|2020-01-11 04:12||aol_gold_support||Note Edited: 0036040||View Revisions|
|2020-01-11 04:12||UbitUmarov||Note Deleted: 0036040|
|Copyright © 2000 - 2012 MantisBT Group|