Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008630opensim[REGION] Scripting Enginepublic2019-11-21 04:352019-11-29 22:37
Reportertampa 
Assigned To 
PriorityurgentSeveritymajorReproducibilityalways
StatusconfirmedResolutionopen 
PlatformOSOS Version
Product Version0.9.1.0 
Target VersionFixed in Version 
Summary0008630: Memory Leak on YEngine
DescriptionI believe this to be caused by dynamic textures not clearing as well, to a smaller part, http requests data sticking around.

Been monitoring a small test region with just a few scripts on them, over a day it has accumulated over 3gb of memory, initially started with 200mb:
https://i.imgur.com/HfoJ2I9.png [^]

On other regions with more dynamic textures and http requests over time the leak is much more pronounced so much so I found a simulator consuming 32gb of memory up top its normal 4gb.

Steps To ReproduceSpin up a simulator under YEngine
Create a script with periodic dynamic textures drawing and http requests
Watch memory usage go up and up and up
Additional InformationI first noticed this after switching from XEngine to YEngine without doing any other changes to the simulator and region, got a monitoring alert because the whole server was out of memory.

There was an old bug with XEngine not clearing dynamic textures very well, that has since been resolved though.
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (1 Region per Sim) , Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Script EngineYEngine
EnvironmentMono / Linux64
Mono Version6.x
Viewer
Attached Files

- Relationships

-  Notes
(0035898)
UbitUmarov (administrator)
2019-11-21 05:01
edited on: 2019-11-21 05:01

dynamic texture code is the same on X and Y. Engines only see UUIDs and not the bulk data, so one can leak if the other doesnt

(0035899)
UbitUmarov (administrator)
2019-11-21 06:10

As tampa and i seen, one of is test scripts shown no leaks on mono 6.0.??

He is using last stable 6.4 also with a new libgdi...
(0035900)
tampa (reporter)
2019-11-21 06:14

I will test with mono 6.0 as well after reverting back to snapshot and building new binaries. If that fixes it then we know it is probably related to changes in 6.4 release which is current stable for mono. This however means regardless it needs looking at because naturally mono website redirects to latest stable release so people will install that.

Assuming either changes to gc or libgdiplus are the culprits, but as Ubit said figuring that out might be very hard :S

Will report back with what I find on mono 6.0, for now I guess a careful warning not to use 6.4 in the meantime(add to wiki perhaps?)
(0035901)
tampa (reporter)
2019-11-21 06:52

So the issue can be traced to libgdiplus. After installing the distro version from ubuntu mirrors the problem went away.

apt purge libgdiplus
apt install libgdiplus=4.2-1ubuntu1

This means whatever changed in libgdiplus 6.0.4 is the cause of the leak under YEngine, meaning then either sticking to old version of libgdiplus or fixing what's broken :/
(0035902)
piusnoel (reporter)
2019-11-21 08:37

I created 4 cubes with the scripts tampa posted in a link on #opensim-dev@freenode yesterday on a simulator running XEngine and a simulator running YEngine.

In a first run over night (roughly 9 hours) I started only one of the four scripts on the simulator with XEngine, but I've seen a systemwide increase of 7 GB.

Next I started a second run where I started all four scripts on both engines and monitored each simulator separately. After 6 hours it summed up to an identical increase of roughly 2.5 GB on each simulator.

I tested on Ubuntu 18.04 with Mono 6.4.0.198 with libgdiplus 6.0.4.

P.S. Currently there is an open pull request related to resource leaks on https://github.com/mono/libgdiplus. [^] Not sure this will resolve the issue in the furture.
(0035903)
piusnoel (reporter)
2019-11-22 02:49

It looks ok after testing 14 hours on Ubuntu 18.04 running Mono 6.4.0.198 with libgdiplus 4.2-2. The problem definitely seems to be in libgdiplus.
(0035904)
tampa (reporter)
2019-11-22 05:23

Confirming this for now, though whether this warrants changes or just some time to wait for updates on libgdiplus is in the stars
(0035915)
tampa (reporter)
2019-11-28 05:39

Apparently there were some updates to libgdiplus recently that may help resolve this issue, however no new release has been made yet so these changes only apply if built from their master. Since it then also depends on mono to update to that latest version, seeing as distros are unlikely to do so, it may be even longer until an update resolves this.

I propose we resolve this temporarily by shipping our own libgdiplus along with either bin or libomv and only load a different version if the version on the system is newer than our own, that way we can either ship an older working version or the latest from their master to hopefully solve this before every other simulator starts choking on textures.

Else I feel the only way is a big red warning on the wiki regarding the install of mono requiring the use of older versions of libgdiplus :/

- Issue History
Date Modified Username Field Change
2019-11-21 04:35 tampa New Issue
2019-11-21 05:01 UbitUmarov Note Added: 0035898
2019-11-21 05:01 UbitUmarov Note Edited: 0035898 View Revisions
2019-11-21 06:10 UbitUmarov Note Added: 0035899
2019-11-21 06:14 tampa Note Added: 0035900
2019-11-21 06:52 tampa Note Added: 0035901
2019-11-21 08:37 piusnoel Note Added: 0035902
2019-11-22 02:49 piusnoel Note Added: 0035903
2019-11-22 05:23 tampa Note Added: 0035904
2019-11-22 05:23 tampa Status new => confirmed
2019-11-28 05:39 tampa Note Added: 0035915
2019-11-29 22:32 geektech12 Note Added: 0035916
2019-11-29 22:34 geektech12 Note Edited: 0035916 View Revisions
2019-11-29 22:34 UbitUmarov Note Deleted: 0035916
2019-11-29 22:37 geektech12 Note Added: 0035917
2019-11-29 22:37 geektech12 Note Edited: 0035917 View Revisions
2019-11-29 23:43 UbitUmarov Note Deleted: 0035917


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker