MantisBT - opensim
View Issue Details
0006548opensim[REGION] OpenSim Corepublic2013-02-23 08:062014-07-29 13:43
Snoopy 
Snoopy 
urgentmajoralways
closedfixed 
Linux, 64 bitUbuntu12.04.2
master (dev code) 
master (dev code) 
Grid (Multiple Regions per Sim)
ODE
Mono / Linux64
2.10
0006548: Very High Memory Consumption
An OpenSim region process running 2 regions shows a dramatic increase of memory consumption as soon as more than about 4 avatars visit one of the region.

With 10 to 15 avatars the initial memory consumption increases by more than 1 GB memory. Previous OpenSim versions did show a by far lower increase of memory consumption with that number of visiting avatars.
No tags attached.
related to 0006570closed Teravus [RemoteAdmin] responce is delayed in every 2nd call 
Issue History
2013-02-23 08:06SnoopyNew Issue
2013-02-23 08:07SnoopyNote Added: 0023594
2013-02-25 02:24SnoopyNote Added: 0023596
2013-02-26 16:36justinccNote Added: 0023608
2013-02-27 09:01SnoopyNote Added: 0023611
2013-02-27 12:48justinccNote Added: 0023613
2013-03-10 14:07SnoopyRelationship addedrelated to 0006570
2013-03-12 11:02Michelle ArgusNote Added: 0023657
2013-03-26 17:51justinccNote Added: 0023700
2013-03-27 12:44SnoopyNote Added: 0023705
2013-04-03 17:58justinccNote Added: 0023748
2014-04-16 03:02SnoopyStatusnew => resolved
2014-04-16 03:02SnoopyResolutionopen => fixed
2014-04-16 03:02SnoopyAssigned To => Snoopy
2014-07-29 13:43chi11kenStatusresolved => closed

Notes
(0023594)
Snoopy   
2013-02-23 08:07   
By the way, that is the exact version used:
  OpenSim 0.7.6 Dev 6f3dcf58b8f47922b68993cfdd9cdbd5b4ae36d8 r/22031
(0023596)
Snoopy   
2013-02-25 02:24   
When users leave the region not all memory seems to be frees, so there seems to be some serious memory leak in the newer OpenSim version.

Beside that I found out that (some?) RemoteAdmin calls to not return a result anymore, although the correspronding RemoteAdmin code did not change. Is it possible that there is some bug in the basic communications handlers, which also could be a reason for the memory leak described in this report?
(0023608)
justincc   
2013-02-26 16:36   
What memory figure are you using?

Any issue is very unlikely to be related to RemoteAdmin call failures. I don't know why these would have stopped working.
(0023611)
Snoopy   
2013-02-27 09:01   
> What memory figure are you using?

The total memory consumed by that OpenSim process.

> Any issue is very unlikely to be related to RemoteAdmin call failures. I don't know why these would have stopped working.

Try update_user, for example to just change the password. That RemoteAdmin call succeeds, but never returns the success code. Instead a timeout happens on the calleds side after waiting a long time.
(0023613)
justincc   
2013-02-27 12:48   
At least on Mono, when memory is allocated to the process it is never released, even if help memory usage declines. You are better off look at the internal "Allocated to OpenSim objects" measure, though this can fluctuate a lot if the region has high memory churn.

As for admin_update_user, there's a log notice way back from Crista on 2010-01-08 (b63405c1) that this needed to be updated for 0.7, so I have no idea if it has been working at all. Unfortunately, I do not currently have time to look and nobody raised that bug during the release candidate process.
(0023657)
Michelle Argus   
2013-03-12 11:02   
Well, I can confirm that the RemoteAdmin is having issues. These issues started in some version since december. Every 2nd call to remote admin does not fire the module commands (or only after a long time, >1 min). This affects all commands I have tested. I also have a own module based on remote admin which also has the same issues.


However, I am not seeing noticable memory increase which might mean that the RemoteAdmin issues are not related
(0023700)
justincc   
2013-03-26 17:51   
I ran into the RemoteAdmin issue myself today with the PHP functions file_get_contents() and stream_get_contents().

It appears that unlike web-browsers and curl, these PHP functions wait for the TCP connection to be closed before returning [1]. And OpenSimulator is not doing this for a while on some requests, even if Connection: close is specified as a request header from the PHP. Needs further investigation, but in the meantime I would suggest trying curl.

[1] http://www.php.net/manual/en/function.file-get-contents.php#101026 [^]
(0023705)
Snoopy   
2013-03-27 12:44   
There is a similar problem with requests sent to other OpenSim processes using the [FORMS] module. For example this is used by the groups module. The request to check the presence of all group members hangs until a timeout produces such error messages after a while:

INFO - OpenSim.Framework.SynchronousRestFormsRequester [FORMS]: Slow request 529 POST http://presence.xyz.com:8003/presence [^] took 99999ms, 99999ms writing, VERSIONMIN=0&VERSIONMAX=0&METHOD=getagents&uuids[]=bcc9a5b6-bcfc-4e02-be2d-fb759aae9def

There seems to b serious problems with streams not getting closed in the latest OpenSim versions. These cause these RemoteAdmin problems as well as issues with the groups module.
(0023748)
justincc   
2013-04-03 17:58   
Failing commit for RemoteAdmin issue appears to be 1dc09d8e8f4a6caa321d0227722af97ee4aeed6a (Tue Feb 5 18:02:25 2013) according to git bisect.