|Anonymous | Login | Signup for a new account||2020-10-24 16:25 PDT|
|Main | My View | View Issues | Change Log | Roadmap | Summary | My Account|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0008630||opensim||[REGION] Scripting Engine||public||2019-11-21 04:35||2020-08-01 08:17|
|Status||resolved||Resolution||no change required|
|Target Version||Fixed in Version||master (dev code)|
|Summary||0008630: Memory Leak on YEngine|
|Description||I believe this to be caused by dynamic textures not clearing as well, to a smaller part, http requests data sticking around.|
Been monitoring a small test region with just a few scripts on them, over a day it has accumulated over 3gb of memory, initially started with 200mb:
On other regions with more dynamic textures and http requests over time the leak is much more pronounced so much so I found a simulator consuming 32gb of memory up top its normal 4gb.
|Steps To Reproduce||Spin up a simulator under YEngine|
Create a script with periodic dynamic textures drawing and http requests
Watch memory usage go up and up and up
|Additional Information||I first noticed this after switching from XEngine to YEngine without doing any other changes to the simulator and region, got a monitoring alert because the whole server was out of memory.|
There was an old bug with XEngine not clearing dynamic textures very well, that has since been resolved though.
|Tags||No tags attached.|
|Git Revision or version number|
|Run Mode||Grid (1 Region per Sim) , Grid (Multiple Regions per Sim)|
|Environment||Mono / Linux64|
edited on: 2019-11-21 05:01
dynamic texture code is the same on X and Y. Engines only see UUIDs and not the bulk data, so one can leak if the other doesnt
As tampa and i seen, one of is test scripts shown no leaks on mono 6.0.??
He is using last stable 6.4 also with a new libgdi...
I will test with mono 6.0 as well after reverting back to snapshot and building new binaries. If that fixes it then we know it is probably related to changes in 6.4 release which is current stable for mono. This however means regardless it needs looking at because naturally mono website redirects to latest stable release so people will install that.
Assuming either changes to gc or libgdiplus are the culprits, but as Ubit said figuring that out might be very hard :S
Will report back with what I find on mono 6.0, for now I guess a careful warning not to use 6.4 in the meantime(add to wiki perhaps?)
So the issue can be traced to libgdiplus. After installing the distro version from ubuntu mirrors the problem went away.
apt purge libgdiplus
apt install libgdiplus=4.2-1ubuntu1
This means whatever changed in libgdiplus 6.0.4 is the cause of the leak under YEngine, meaning then either sticking to old version of libgdiplus or fixing what's broken :/
I created 4 cubes with the scripts tampa posted in a link on #opensim-dev@freenode yesterday on a simulator running XEngine and a simulator running YEngine.
In a first run over night (roughly 9 hours) I started only one of the four scripts on the simulator with XEngine, but I've seen a systemwide increase of 7 GB.
Next I started a second run where I started all four scripts on both engines and monitored each simulator separately. After 6 hours it summed up to an identical increase of roughly 2.5 GB on each simulator.
I tested on Ubuntu 18.04 with Mono 126.96.36.199 with libgdiplus 6.0.4.
P.S. Currently there is an open pull request related to resource leaks on https://github.com/mono/libgdiplus. [^] Not sure this will resolve the issue in the furture.
|It looks ok after testing 14 hours on Ubuntu 18.04 running Mono 188.8.131.52 with libgdiplus 4.2-2. The problem definitely seems to be in libgdiplus.|
|Confirming this for now, though whether this warrants changes or just some time to wait for updates on libgdiplus is in the stars|
Apparently there were some updates to libgdiplus recently that may help resolve this issue, however no new release has been made yet so these changes only apply if built from their master. Since it then also depends on mono to update to that latest version, seeing as distros are unlikely to do so, it may be even longer until an update resolves this.
I propose we resolve this temporarily by shipping our own libgdiplus along with either bin or libomv and only load a different version if the version on the system is newer than our own, that way we can either ship an older working version or the latest from their master to hopefully solve this before every other simulator starts choking on textures.
Else I feel the only way is a big red warning on the wiki regarding the install of mono requiring the use of older versions of libgdiplus :/
There has not been a new release of libgdiplus and the version shipped with mono stable branch is still the same containing this massive memory leak. I have attempted to call attention to this issue to the mono project, but got no response.
It is possible to install an older version, usually 4.2 on most linux distros as that is the default shipped, but it could be that upcoming releases of those distros may not even ship with the older version anymore.
This is a big problem for regions with dynamic texture generation which is common now. It is not an easy task, but a solution needs to be found, preferably before it becomes impossible to use a working version of the lib.
Libgdiplus has been updated to version 6.0.5 in mono 6.10
Currently this is available through the preview builds of mono, but the next stable will likely contain this new version.
From testing it seems to solve the biggest part of the memory leak regarding dynamic textures. There is still a few mb that leak over time, but within 24 hours that only amounts to about 50mb total.
So it looks like this will resolve itself once mono stable is on 6.10 which will likely happen this year. Preview build seems stable and even comes with some gc improvements so for anyone currently using the distro version of libgdiplus I would advice to simply switch to preview mono builds until stable release.
With mono 6.10 released as stable and libgdiplus updated along with it the major memory leaking seems to be resolved. According to their issue tickets there might still be something in mono itself, but I suspect that is not as major as I have not seen any runaway behavior with the latest release.
I suggest either use the latest stable, verion 6.10 as of this post, or an older version of mono that still uses the older libgdiplus version 4.x, though it is probably best for compatibility to use newer.
|2019-11-21 04:35||tampa||New Issue|
|2019-11-21 05:01||UbitUmarov||Note Added: 0035898|
|2019-11-21 05:01||UbitUmarov||Note Edited: 0035898||View Revisions|
|2019-11-21 06:10||UbitUmarov||Note Added: 0035899|
|2019-11-21 06:14||tampa||Note Added: 0035900|
|2019-11-21 06:52||tampa||Note Added: 0035901|
|2019-11-21 08:37||piusnoel||Note Added: 0035902|
|2019-11-22 02:49||piusnoel||Note Added: 0035903|
|2019-11-22 05:23||tampa||Note Added: 0035904|
|2019-11-22 05:23||tampa||Status||new => confirmed|
|2019-11-28 05:39||tampa||Note Added: 0035915|
|2019-11-29 22:32||geektech12||Note Added: 0035916|
|2019-11-29 22:34||geektech12||Note Edited: 0035916||View Revisions|
|2019-11-29 22:34||UbitUmarov||Note Deleted: 0035916|
|2019-11-29 22:37||geektech12||Note Added: 0035917|
|2019-11-29 22:37||geektech12||Note Edited: 0035917||View Revisions|
|2019-11-29 23:43||UbitUmarov||Note Deleted: 0035917|
|2020-01-11 04:11||aol_gold_support||Note Added: 0036040|
|2020-01-11 04:12||aol_gold_support||Note Edited: 0036040||View Revisions|
|2020-01-11 04:12||UbitUmarov||Note Deleted: 0036040|
|2020-03-19 08:49||tampa||Note Added: 0036294|
|2020-05-03 15:32||tampa||Note Added: 0036441|
|2020-08-01 08:17||tampa||Note Added: 0036646|
|2020-08-01 08:17||tampa||Status||confirmed => resolved|
|2020-08-01 08:17||tampa||Fixed in Version||=> master (dev code)|
|2020-08-01 08:17||tampa||Resolution||open => no change required|
|2020-08-01 08:17||tampa||Assigned To||=> tampa|
|Copyright © 2000 - 2012 MantisBT Group|