Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008630opensim[REGION] Scripting Enginepublic2019-11-21 04:352020-08-01 08:17
Assigned Totampa 
StatusresolvedResolutionno change required 
PlatformOSOS Version
Product Version0.9.1.0 
Target VersionFixed in Versionmaster (dev code) 
Summary0008630: Memory Leak on YEngine
DescriptionI believe this to be caused by dynamic textures not clearing as well, to a smaller part, http requests data sticking around.

Been monitoring a small test region with just a few scripts on them, over a day it has accumulated over 3gb of memory, initially started with 200mb: [^]

On other regions with more dynamic textures and http requests over time the leak is much more pronounced so much so I found a simulator consuming 32gb of memory up top its normal 4gb.

Steps To ReproduceSpin up a simulator under YEngine
Create a script with periodic dynamic textures drawing and http requests
Watch memory usage go up and up and up
Additional InformationI first noticed this after switching from XEngine to YEngine without doing any other changes to the simulator and region, got a monitoring alert because the whole server was out of memory.

There was an old bug with XEngine not clearing dynamic textures very well, that has since been resolved though.
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (1 Region per Sim) , Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Script EngineYEngine
EnvironmentMono / Linux64
Mono Version6.x
Attached Files

- Relationships

-  Notes
UbitUmarov (administrator)
2019-11-21 05:01
edited on: 2019-11-21 05:01

dynamic texture code is the same on X and Y. Engines only see UUIDs and not the bulk data, so one can leak if the other doesnt

UbitUmarov (administrator)
2019-11-21 06:10

As tampa and i seen, one of is test scripts shown no leaks on mono 6.0.??

He is using last stable 6.4 also with a new libgdi...
tampa (reporter)
2019-11-21 06:14

I will test with mono 6.0 as well after reverting back to snapshot and building new binaries. If that fixes it then we know it is probably related to changes in 6.4 release which is current stable for mono. This however means regardless it needs looking at because naturally mono website redirects to latest stable release so people will install that.

Assuming either changes to gc or libgdiplus are the culprits, but as Ubit said figuring that out might be very hard :S

Will report back with what I find on mono 6.0, for now I guess a careful warning not to use 6.4 in the meantime(add to wiki perhaps?)
tampa (reporter)
2019-11-21 06:52

So the issue can be traced to libgdiplus. After installing the distro version from ubuntu mirrors the problem went away.

apt purge libgdiplus
apt install libgdiplus=4.2-1ubuntu1

This means whatever changed in libgdiplus 6.0.4 is the cause of the leak under YEngine, meaning then either sticking to old version of libgdiplus or fixing what's broken :/
piusnoel (reporter)
2019-11-21 08:37

I created 4 cubes with the scripts tampa posted in a link on #opensim-dev@freenode yesterday on a simulator running XEngine and a simulator running YEngine.

In a first run over night (roughly 9 hours) I started only one of the four scripts on the simulator with XEngine, but I've seen a systemwide increase of 7 GB.

Next I started a second run where I started all four scripts on both engines and monitored each simulator separately. After 6 hours it summed up to an identical increase of roughly 2.5 GB on each simulator.

I tested on Ubuntu 18.04 with Mono with libgdiplus 6.0.4.

P.S. Currently there is an open pull request related to resource leaks on [^] Not sure this will resolve the issue in the furture.
piusnoel (reporter)
2019-11-22 02:49

It looks ok after testing 14 hours on Ubuntu 18.04 running Mono with libgdiplus 4.2-2. The problem definitely seems to be in libgdiplus.
tampa (reporter)
2019-11-22 05:23

Confirming this for now, though whether this warrants changes or just some time to wait for updates on libgdiplus is in the stars
tampa (reporter)
2019-11-28 05:39

Apparently there were some updates to libgdiplus recently that may help resolve this issue, however no new release has been made yet so these changes only apply if built from their master. Since it then also depends on mono to update to that latest version, seeing as distros are unlikely to do so, it may be even longer until an update resolves this.

I propose we resolve this temporarily by shipping our own libgdiplus along with either bin or libomv and only load a different version if the version on the system is newer than our own, that way we can either ship an older working version or the latest from their master to hopefully solve this before every other simulator starts choking on textures.

Else I feel the only way is a big red warning on the wiki regarding the install of mono requiring the use of older versions of libgdiplus :/
tampa (reporter)
2020-03-19 08:49

There has not been a new release of libgdiplus and the version shipped with mono stable branch is still the same containing this massive memory leak. I have attempted to call attention to this issue to the mono project, but got no response.

It is possible to install an older version, usually 4.2 on most linux distros as that is the default shipped, but it could be that upcoming releases of those distros may not even ship with the older version anymore.

This is a big problem for regions with dynamic texture generation which is common now. It is not an easy task, but a solution needs to be found, preferably before it becomes impossible to use a working version of the lib.
tampa (reporter)
2020-05-03 15:32

Libgdiplus has been updated to version 6.0.5 in mono 6.10
Currently this is available through the preview builds of mono, but the next stable will likely contain this new version.

From testing it seems to solve the biggest part of the memory leak regarding dynamic textures. There is still a few mb that leak over time, but within 24 hours that only amounts to about 50mb total.

So it looks like this will resolve itself once mono stable is on 6.10 which will likely happen this year. Preview build seems stable and even comes with some gc improvements so for anyone currently using the distro version of libgdiplus I would advice to simply switch to preview mono builds until stable release.
tampa (reporter)
2020-08-01 08:17

With mono 6.10 released as stable and libgdiplus updated along with it the major memory leaking seems to be resolved. According to their issue tickets there might still be something in mono itself, but I suspect that is not as major as I have not seen any runaway behavior with the latest release.

I suggest either use the latest stable, verion 6.10 as of this post, or an older version of mono that still uses the older libgdiplus version 4.x, though it is probably best for compatibility to use newer.

- Issue History
Date Modified Username Field Change
2019-11-21 04:35 tampa New Issue
2019-11-21 05:01 UbitUmarov Note Added: 0035898
2019-11-21 05:01 UbitUmarov Note Edited: 0035898 View Revisions
2019-11-21 06:10 UbitUmarov Note Added: 0035899
2019-11-21 06:14 tampa Note Added: 0035900
2019-11-21 06:52 tampa Note Added: 0035901
2019-11-21 08:37 piusnoel Note Added: 0035902
2019-11-22 02:49 piusnoel Note Added: 0035903
2019-11-22 05:23 tampa Note Added: 0035904
2019-11-22 05:23 tampa Status new => confirmed
2019-11-28 05:39 tampa Note Added: 0035915
2019-11-29 22:32 geektech12 Note Added: 0035916
2019-11-29 22:34 geektech12 Note Edited: 0035916 View Revisions
2019-11-29 22:34 UbitUmarov Note Deleted: 0035916
2019-11-29 22:37 geektech12 Note Added: 0035917
2019-11-29 22:37 geektech12 Note Edited: 0035917 View Revisions
2019-11-29 23:43 UbitUmarov Note Deleted: 0035917
2020-01-11 04:11 aol_gold_support Note Added: 0036040
2020-01-11 04:12 aol_gold_support Note Edited: 0036040 View Revisions
2020-01-11 04:12 UbitUmarov Note Deleted: 0036040
2020-03-19 08:49 tampa Note Added: 0036294
2020-05-03 15:32 tampa Note Added: 0036441
2020-08-01 08:17 tampa Note Added: 0036646
2020-08-01 08:17 tampa Status confirmed => resolved
2020-08-01 08:17 tampa Fixed in Version => master (dev code)
2020-08-01 08:17 tampa Resolution open => no change required
2020-08-01 08:17 tampa Assigned To => tampa

Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker