Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006548opensim[REGION] OpenSim Corepublic2013-02-23 08:062014-07-29 13:43
ReporterSnoopy 
Assigned ToSnoopy 
PriorityurgentSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformLinux, 64 bitOSUbuntuOS Version12.04.2
Product Versionmaster (dev code) 
Target Versionmaster (dev code)Fixed in Version 
Summary0006548: Very High Memory Consumption
DescriptionAn OpenSim region process running 2 regions shows a dramatic increase of memory consumption as soon as more than about 4 avatars visit one of the region.

With 10 to 15 avatars the initial memory consumption increases by more than 1 GB memory. Previous OpenSim versions did show a by far lower increase of memory consumption with that number of visiting avatars.
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (Multiple Regions per Sim)
Physics EngineODE
EnvironmentMono / Linux64
Mono Version2.10
Viewer
Attached Files

- Relationships
related to 0006570closedTeravus [RemoteAdmin] responce is delayed in every 2nd call 

-  Notes
(0023594)
Snoopy (administrator)
2013-02-23 08:07

By the way, that is the exact version used:
  OpenSim 0.7.6 Dev 6f3dcf58b8f47922b68993cfdd9cdbd5b4ae36d8 r/22031
(0023596)
Snoopy (administrator)
2013-02-25 02:24

When users leave the region not all memory seems to be frees, so there seems to be some serious memory leak in the newer OpenSim version.

Beside that I found out that (some?) RemoteAdmin calls to not return a result anymore, although the correspronding RemoteAdmin code did not change. Is it possible that there is some bug in the basic communications handlers, which also could be a reason for the memory leak described in this report?
(0023608)
justincc (administrator)
2013-02-26 16:36

What memory figure are you using?

Any issue is very unlikely to be related to RemoteAdmin call failures. I don't know why these would have stopped working.
(0023611)
Snoopy (administrator)
2013-02-27 09:01

> What memory figure are you using?

The total memory consumed by that OpenSim process.

> Any issue is very unlikely to be related to RemoteAdmin call failures. I don't know why these would have stopped working.

Try update_user, for example to just change the password. That RemoteAdmin call succeeds, but never returns the success code. Instead a timeout happens on the calleds side after waiting a long time.
(0023613)
justincc (administrator)
2013-02-27 12:48

At least on Mono, when memory is allocated to the process it is never released, even if help memory usage declines. You are better off look at the internal "Allocated to OpenSim objects" measure, though this can fluctuate a lot if the region has high memory churn.

As for admin_update_user, there's a log notice way back from Crista on 2010-01-08 (b63405c1) that this needed to be updated for 0.7, so I have no idea if it has been working at all. Unfortunately, I do not currently have time to look and nobody raised that bug during the release candidate process.
(0023657)
Michelle Argus (reporter)
2013-03-12 11:02

Well, I can confirm that the RemoteAdmin is having issues. These issues started in some version since december. Every 2nd call to remote admin does not fire the module commands (or only after a long time, >1 min). This affects all commands I have tested. I also have a own module based on remote admin which also has the same issues.


However, I am not seeing noticable memory increase which might mean that the RemoteAdmin issues are not related
(0023700)
justincc (administrator)
2013-03-26 17:51

I ran into the RemoteAdmin issue myself today with the PHP functions file_get_contents() and stream_get_contents().

It appears that unlike web-browsers and curl, these PHP functions wait for the TCP connection to be closed before returning [1]. And OpenSimulator is not doing this for a while on some requests, even if Connection: close is specified as a request header from the PHP. Needs further investigation, but in the meantime I would suggest trying curl.

[1] http://www.php.net/manual/en/function.file-get-contents.php#101026 [^]
(0023705)
Snoopy (administrator)
2013-03-27 12:44

There is a similar problem with requests sent to other OpenSim processes using the [FORMS] module. For example this is used by the groups module. The request to check the presence of all group members hangs until a timeout produces such error messages after a while:

INFO - OpenSim.Framework.SynchronousRestFormsRequester [FORMS]: Slow request 529 POST http://presence.xyz.com:8003/presence [^] took 99999ms, 99999ms writing, VERSIONMIN=0&VERSIONMAX=0&METHOD=getagents&uuids[]=bcc9a5b6-bcfc-4e02-be2d-fb759aae9def

There seems to b serious problems with streams not getting closed in the latest OpenSim versions. These cause these RemoteAdmin problems as well as issues with the groups module.
(0023748)
justincc (administrator)
2013-04-03 17:58

Failing commit for RemoteAdmin issue appears to be 1dc09d8e8f4a6caa321d0227722af97ee4aeed6a (Tue Feb 5 18:02:25 2013) according to git bisect.

- Issue History
Date Modified Username Field Change
2013-02-23 08:06 Snoopy New Issue
2013-02-23 08:07 Snoopy Note Added: 0023594
2013-02-25 02:24 Snoopy Note Added: 0023596
2013-02-26 16:36 justincc Note Added: 0023608
2013-02-27 09:01 Snoopy Note Added: 0023611
2013-02-27 12:48 justincc Note Added: 0023613
2013-03-10 14:07 Snoopy Relationship added related to 0006570
2013-03-12 11:02 Michelle Argus Note Added: 0023657
2013-03-26 17:51 justincc Note Added: 0023700
2013-03-27 12:44 Snoopy Note Added: 0023705
2013-04-03 17:58 justincc Note Added: 0023748
2014-04-16 03:02 Snoopy Status new => resolved
2014-04-16 03:02 Snoopy Resolution open => fixed
2014-04-16 03:02 Snoopy Assigned To => Snoopy
2014-07-29 13:43 chi11ken Status resolved => closed


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker