Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0002688opensim[REGION] OpenSim Corepublic2008-11-25 08:352011-12-09 16:41
ReporterHell Fire 
Assigned To 
PrioritynormalSeveritymajorReproducibilityalways
StatusacknowledgedResolutionopen 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0002688: Triggering high memory use in sim with low prim/script count
DescriptionSpent a little time tonight (GMT+10) testing different situations on which opensim chews through memory.

This one I'm not sure what side is the cause, object itself or the scripts. Anyway, here it is, two objects, a spawner and a dummy that just has an llDie() on a timer event.

Set up the Scripted Die object, dump it in the spawner, spawner tries to do llRezObject every second, Scripted Die object kills itself after a couple of minutes, keeps the prim and script count down, but memory use still rockets up.

With the 1 second rez loop and 120 second die timer, prims count on my sim never gets above 170.

Memory use starts off at about 120MB for the mostly empty sim (1 root avi, 50ish prims), after the first two minutes it's at ~500MB, 5 minutes ~700MB, 10 minutes ~800MB (starting to use 100% of a CPU core at this point), at 15 minutes 870MB used, and the sim starts to freeze up, start receiving:

01:10:35 - [ScriptEngine.DotNetEngine]: ERROR: Event execution queue item count is at 300. Config variable "EventExecutionMaxQueueSize" is set to 300, so ignoring new event.
01:10:35 - [ScriptEngine.DotNetEngine]: Event ignored: localID: 720036, itemID: c3ff7fe1-c88a-4912-a31a-552ae5c04dfb, FunctionName: timer

Guess the sim starts to lag running out of CPU from the GC running constantly and causes the event execution queue of both the spawner and the llDie() blocks to overflow.

If you stop the spammer object before the GC lags the sim out, the sim never recovers. Stopping the spammer on my sim and waiting for the dummys to die leaves the memory use up at ~720MB (and slowly creeping).

Originally tested with Mono 1.9.1, but updated to check with 2.0.1 as I had read an archived email that the Mono team were tweaking the GC in 2.0.

Running SVN 7463 - Mono JIT compiler version 2.0.1 (tarball) - Gentoo Base System release 1.12.11.1 - Linux rackfire 2.6.25-gentoo-r7 0000003 SMP Sun Sep 21 00:02:48 EST 2008 x86_64 Dual Core AMD Opteron(tm) Processor 265 AuthenticAMD GNU/Linux
Additional InformationSpawner Script:
default
{
    state_entry() {
        llSetText("Spammer Prim - Stress Testing/Memory Leak finding", <1.0,1.0,1.0>, 0.8);
        llListen(0, "", llGetOwner(), "!startspam");
        llListen(0, "", llGetOwner(), "!stopspam");
        llOwnerSay("Commands: !startspam !stopspam");
    }
    
    listen(integer chan, string name, key id, string msg) {
        if (msg == "!startspam") {
            llOwnerSay("Starting spam");
            llSetTimerEvent(1);
        } else {
            llOwnerSay("Stopping spam");
            llSetTimerEvent(0);
        }
    }
    
    timer() {
        llRezObject("Scripted Die in 2 minutes", llGetPos() + <0.0,0.0,2.0>, ZERO_VECTOR, llGetRot(), FALSE);
    }
}



"Scripted Die in 2 minutes" object script:
default {
    state_entry() {
        llSetTimerEvent(120);
    }
    
    timer() {
        llDie();
    }
}
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (1 Region per Sim)
Physics EngineODE
Script Engine
EnvironmentMono / Linux64
Mono Version
Viewer
Attached Files

- Relationships

-  Notes
(0007321)
Hell Fire (reporter)
2008-11-25 09:17
edited on: 2008-11-25 09:19

Switched to the 32 bit chroot, updated, rebuilt, switched ODE to 32 bit, another different version of Mono under the chroot:
Running SVN 7465 - Mono JIT compiler version 1.2.5.1 (tarball) - Gentoo Base System release 1.12.11.1 - Linux rackfire 2.6.25-gentoo-r7 0000003 SMP Sun Sep 21 00:02:48 EST 2008 i686 Dual Core AMD Opteron(tm) Processor 265 AuthenticAMD GNU/Linux

Similar results (at about half the usage on the "leak"):
Base Usage: 100MB
1m: 190MB
2m: 280MB
3m: 300MB
4m: 330MB
5m: 345MB
6m: 365MB
7m: 390MB
8m: 400MB
9m: 420MB
10m: 440MB

Ed: forgot to mention, DotNetEngine for the script engine

(0007333)
Hell Fire (reporter)
2008-11-25 17:24

Little more testing again, this time changing the script engine up to XEngine. Nowhere near the excessively large memory jumps but it still creeps in usage.

Changed the dummy object to have a 10 second llDie timer, copied the spammer script into the spammer object 10 times (so 10 llRezObjects a second).

Running SVN 7473 - Mono JIT compiler version 2.0.1 (tarball) - Gentoo Base System release 1.12.11.1 - Linux rackfire 2.6.25-gentoo-r7 0000003 SMP Sun Sep 21 00:02:48 EST 2008 x86_64 Dual Core AMD Opteron(tm) Processor 265 AuthenticAMD GNU/Linux

In about 5 minutes, memory usage was up from 110MB to 200MB, after the spawner is told to stop, it didn't drop back down.

So, the huge jumps before were from DotNetEngine, but switching to XEngine, while there's no more huge jumps in usage, still creeps up slowly.

Might test more tonight, work calls.
(0007338)
Adam (administrator)
2008-11-25 19:48

Thankyou!

I had not considered doing something like this for tracking down memory leaks. This is a good idea - I'll try get someone to run this under a memory profiler and see what we get left over with.
(0007342)
tedd (manager)
2008-11-26 04:09

I have changed how script object lease is handled inside AppDomain. Scripts should now loose their lifetime lease when a script is unloaded. The assembly will still occupy space until AppDomain is unloaded, but the script instance data should be wiped.

Patch in r2465 and r2466
(0007343)
Hell Fire (reporter)
2008-11-26 04:52
edited on: 2008-11-26 05:08

On loading of the sim, happens once per script instance (XEngine):

22:49:12 - [XEngine]: Loaded script Spammer Prim.Spammer 6
22:49:12 - [Compiler]: Returning existing assembly for f2dad7e8-5005-4f4b-8cb8-bff18bda4ed8
22:49:12 - [Script] Error loading assembly ScriptEngines/fa02a5e7-8c4c-4042-9315-4016fded5d82/CommonCompiler_compiled_f2dad7e8-5005-4f4b-8cb8-bff18bda4ed8.dll
System.Runtime.Serialization.SerializationException: Type OpenSim.Region.ScriptEngine.Shared.Api.Runtime.ScriptSponsor is not marked as Serializable.
  at System.Runtime.Serialization.Formatters.Binary.BinaryCommon.CheckSerializable (System.Type type, ISurrogateSelector selector, StreamingContext context) [0x0002c] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/BinaryCommon.cs:118
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.GetObjectData (System.Object obj, System.Runtime.Serialization.Formatters.Binary.TypeMetadata& metadata, System.Object& data) [0x00054] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectWriter.cs:386
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteObject (System.IO.BinaryWriter writer, Int64 id, System.Object obj) [0x00000] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectWriter.cs:306
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteObjectInstance (System.IO.BinaryWriter writer, System.Object obj, Boolean isValueObject) [0x00062] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectWriter.cs:293
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteQueuedObjects (System.IO.BinaryWriter writer) [0x00005] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectWriter.cs:271
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteObjectGraph (System.IO.BinaryWriter writer, System.Object obj, System.Runtime.Remoting.Messaging.Header[] headers) [0x0001f] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectWriter.cs:256
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream serializationStream, System.Object graph, System.Runtime.Remoting.Messaging.Header[] headers) [0x000a4] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/BinaryFormatter.cs:230
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream serializationStream, System.Object graph) [0x00000] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/BinaryFormatter.cs:209
  at System.Runtime.Remoting.RemotingServices.SerializeCallData (System.Object obj) [0x00032] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Remoting/RemotingServices.cs:782
  at (wrapper xdomain-invoke) System.Runtime.Remoting.Lifetime.Lease:Register (System.Runtime.Remoting.Lifetime.ISponsor)
  at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance..ctor (IScriptEngine engine, OpenSim.Region.Environment.Scenes.SceneObjectPart part, UUID itemID, UUID assetID, System.String assembly, System.AppDomain dom, System.String primName, System.String scriptName, Int32 startParam, Boolean postOnRez, StateSource stateSource, Int32 maxScriptQueue) [0x0012f] in /mnt/raid1/gentoo32/usr/src/opensim/OpenSim/Region/ScriptEngine/Shared/Instance/ScriptInstance.cs:215
22:49:12 - [Script] Error loading script instance
System.Runtime.Serialization.SerializationException: Type OpenSim.Region.ScriptEngine.Shared.Api.LSL_Api is not marked as Serializable.
  at System.Runtime.Serialization.Formatters.Binary.BinaryCommon.CheckSerializable (System.Type type, ISurrogateSelector selector, StreamingContext context) [0x0002c] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/BinaryCommon.cs:118
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.GetObjectData (System.Object obj, System.Runtime.Serialization.Formatters.Binary.TypeMetadata& metadata, System.Object& data) [0x00054] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectWriter.cs:386
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteObject (System.IO.BinaryWriter writer, Int64 id, System.Object obj) [0x00000] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectWriter.cs:306
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteObjectInstance (System.IO.BinaryWriter writer, System.Object obj, Boolean isValueObject) [0x00062] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectWriter.cs:293
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteQueuedObjects (System.IO.BinaryWriter writer) [0x00005] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectWriter.cs:271
  at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.WriteObjectGraph (System.IO.BinaryWriter writer, System.Object obj, System.Runtime.Remoting.Messaging.Header[] headers) [0x0001f] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectWriter.cs:256
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream serializationStream, System.Object graph, System.Runtime.Remoting.Messaging.Header[] headers) [0x000a4] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/BinaryFormatter.cs:230
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize (System.IO.Stream serializationStream, System.Object graph) [0x00000] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/BinaryFormatter.cs:209
  at System.Runtime.Remoting.RemotingServices.SerializeCallData (System.Object obj) [0x00032] in /var/tmp/portage/dev-lang/mono-2.0.1/work/mono-2.0.1/mcs/class/corlib/System.Runtime.Remoting/RemotingServices.cs:782
  at (wrapper xdomain-invoke) OpenSim.Region.ScriptEngine.Shared.ScriptBase.ScriptBaseClass:InitApi (string,OpenSim.Region.ScriptEngine.Interfaces.IScriptApi)
  at OpenSim.Region.ScriptEngine.Shared.Instance.ScriptInstance..ctor (IScriptEngine engine, OpenSim.Region.Environment.Scenes.SceneObjectPart part, UUID itemID, UUID assetID, System.String assembly, System.AppDomain dom, System.String primName, System.String scriptName, Int32 startParam, Boolean postOnRez, StateSource stateSource, Int32 maxScriptQueue) [0x0018a] in /mnt/raid1/gentoo32/usr/src/opensim/OpenSim/Region/ScriptEngine/Shared/Instance/ScriptInstance.cs:228

(0007344)
Hell Fire (reporter)
2008-11-26 05:08

And:

Spammer Prim: Error compiling script:
Type OpenSim.Region.ScriptEngine.Shared.Api.Runtime.ScriptSponsor is not marked as Serializable.

in DotNetEngine
(0007345)
Hell Fire (reporter)
2008-11-26 05:42

Above is now fixed, changed to:

Spammer Prim: Error compiling script:
Type OpenSim.Region.ScriptEngine.Shared.Api.LSL_Api is not marked as Serializable.
(0007346)
Hell Fire (reporter)
2008-11-26 07:26
edited on: 2008-11-26 07:26

LSL_Api error is now fixed too, scripts compile under both DotNet and XEngine, however scripts stop running after a short time (~ 1:20 from one run, from init of script to unresponsive).

When the script stops responding to events, the timer event keeps firing even though it never gets processed, causing the event execution queue to flood out to max and start dropping the new events.

(0007361)
Hell Fire (reporter)
2008-11-26 16:46
edited on: 2008-11-26 16:47

Alrighty, update to the above, XEngine doesn't flood the Event Execution Queue as such, but when XEngine stops, it seems like the sim stops processing frames.

You can log out and log in, view inventory, load the minimap and worldmap, just about everything except move around or receive objects from the sim. Terrain loads, but not objects.

(Sorry, Rev 7491)

(0020485)
justincc (administrator)
2011-12-09 16:41

Is this still an issue on current OpenSim dev code?

- Issue History
Date Modified Username Field Change
2008-11-25 08:35 Hell Fire New Issue
2008-11-25 08:35 Hell Fire SVN Revision => 7463
2008-11-25 08:35 Hell Fire Run Mode => Grid (1 Region per Sim)
2008-11-25 08:35 Hell Fire Physics Engine => ODE
2008-11-25 08:35 Hell Fire Environment => Mono / Linux64
2008-11-25 09:17 Hell Fire Note Added: 0007321
2008-11-25 09:19 Hell Fire Note Edited: 0007321
2008-11-25 17:24 Hell Fire Note Added: 0007333
2008-11-25 19:48 Adam Note Added: 0007338
2008-11-25 19:49 Adam Status new => acknowledged
2008-11-26 04:09 tedd Note Added: 0007342
2008-11-26 04:52 Hell Fire Note Added: 0007343
2008-11-26 05:08 Hell Fire Note Edited: 0007343
2008-11-26 05:08 Hell Fire Note Added: 0007344
2008-11-26 05:42 Hell Fire Note Added: 0007345
2008-11-26 07:26 Hell Fire Note Added: 0007346
2008-11-26 07:26 Hell Fire Note Edited: 0007346
2008-11-26 16:46 Hell Fire Note Added: 0007361
2008-11-26 16:47 Hell Fire Note Edited: 0007361
2011-12-09 16:41 justincc Note Added: 0020485


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker