[Opensim-users] Intermittent crashing - System.OutofMemoryException

Thu Feb 16 11:53:47 UTC 2012

Cacti will work - 
http://www.cacti.net/downloads/docs/html/install_windows.html

On 02/15/2012 03:24 PM, Justin Clark-Casey wrote:
> Generated errors will not consume memory.
>
> The best way to track usage is to use a system monitoring tool like
> rrdtool to record process memory use (not sure what the Windows
> equivalent is). You can also extract statistics from OpenSim itself [1]
> and record them in the same tool.
>
> I'm not sure if there is any public code on how to glue these things
> together, though I remember that osgrid was recording stats at some point.
>
> If one can record stats then you can see memory usage and starting
> twiddling some config things to try and pin it down. For instance, I
> have reason to be believe that right now, running a large number of
> scripts might leak memory even if no avatars are present in the region.
> I don't know the cause of this, whether it's a general script issue or
> some particular LSL operation.
>
> Unfortunately, this is getting into fairly complex diagnostics but
> running a high uptime simulator with significant usage is not easy. The
> alternative solution is simply to restart the simulator periodically
> (e.g. daily) in order to reset the memory.
>
> [1] http://opensimulator.org/wiki/Monitoring
>
> On 15/02/12 14:58, Rick Anderson wrote:
>> Since one machine is faster than another is there a chance a process
>> that leaks memory is happening faster? Can the memory usage be shown
>> to go to the 2 gig limit then fail? What's a good way to check?
>>
>> Could there be an issue with permissions/settings from OAR import?
>> Like ownership and groups not matching. Then the generated errors
>> consume memory?
>>
>> A couple of random thoughts,
>> -_Rick
>>
>> On Tue, Feb 14, 2012 at 9:50 PM, Justin Clark-Casey
>> <jjustincc at googlemail.com> wrote:
>>> It's a good question. I'm not sure there any such conditions - it
>>> appears
>>> that the time saved by reading this out from memory is dwarfed by other
>>> sources of delay (chiefly network, I expect). All of the 'serious'
>>> simulator operators that I know never go near Flotsam's memory cache.
>>>
>>> James' point about the 2MB limit is a great one - I had forgotten
>>> about this
>>> on Windows. This is the problem that you'll hit first no matter how much
>>> memory you have (which is part of the reason why people run multiple
>>> instances of OpenSimulator rather than a large number of regions on a
>>> single
>>> simulator).
>>>
>>> I believe that it's perfect possible to get Windows to use 64 bit
>>> libraries
>>> when appropriate but this requires doing a bit of grunt work (possibly
>>> compiling 64 bit versions of ode.dll and sqlite3.dll and then putting
>>> the
>>> required detection code into OpenSim).
>>>
>>>
>>> On 14/02/12 15:13, core wrote:
>>>>
>>>> this is an interesting and informative thread. both fleep's and gary's
>>>> comments about flotsam cache settings have made me wonder - exactly
>>>> under what conditions should, if at all, flotsam memory caching be set
>>>> to TRUE. i've tried it both ways and have seen little difference, but
>>>> admittedly this was under moderate usage at best. since there are
>>>> plenty
>>>> of people here running live-fire instances in a variety of environments
>>>> i was hoping to gain some insight.
>>>>
>>>> thanks to hiro for the excellent breakdown of .net memory usage. and to
>>>> fleep, we all know cat hair is in fact magical and makes our gear run
>>>> faster and smarter ;)
>>>>
>>>> - core
>>>>
>>>> On 2/14/2012 7:09 AM, Fleep Tuque wrote:
>>>>>
>>>>> Thanks all for the info and thoughts, sounds like scheduling daily
>>>>> restarts is probably the best option.
>>>>>
>>>>> I'm pretty sure I haven't messed with whatever the default settings
>>>>> are for the flotsam cache, I never copy over old config files when I
>>>>> upgrade, I go through and update by hand and usually just base
>>>>> settings to make sure the upgrade was successful and then turn on
>>>>> other options that we've tested.
>>>>>
>>>>> I have to say, it's almost funny that FleepGrid, which is running
>>>>> Windows XP on a P4 box that was gathering dust and cat hair in the
>>>>> basement, 4GB of RAM, and at least 8 sims full of content and scripts
>>>>> isn't having that problem, but the new poweredge server running
>>>>> WinServer2008, tons of RAM, and only 3 or 4 sims of content is flaking
>>>>> out. In fact, three of the regions on the UCSIM grid are identical to
>>>>> the ones on FleepGrid since I ported them over as OAR files, so it
>>>>> can't be much of a difference in content..
>>>>>
>>>>> Think I'll try going through the log files again, I'd sure like to
>>>>> know what's tipping the out of memory exception!
>>>>>
>>>>> Thanks again.
>>>>>
>>>>> - Chris/Fleep
>>>>>
>>>>> Chris M. Collins (SL/OS: Fleep Tuque)
>>>>> Center for Simulations& Virtual Environments Research (UCSIM)
>>>>> UCIT Instructional& Research Computing
>>>>>
>>>>> University of Cincinnati
>>>>> 406A Zimmer Hall
>>>>> 315 College Drive
>>>>> PO BOX 210088
>>>>> Cincinnati, OH 45221-0088
>>>>> chris.collins at uc.edu<mailto:chris.collins at uc.edu>
>>>>> (513) 556-3018
>>>>>
>>>>> http://ucsim.uc.edu
>>>>>
>>>>> On Tue, Feb 14, 2012 at 7:29 AM, James Stallings II
>>>>> <james.stallings at gmail.com<mailto:james.stallings at gmail.com>> wrote:
>>>>>
>>>>> Regardless the amount of RAM on your system, the .net runtime uses
>>>>> a memory model of a specific size (2 GB) and cannot address more
>>>>> than that except under special circumstances.
>>>>>
>>>>> There is a hack around, involving the use of a Visual Studio C++
>>>>> utility to make a .net binary 'large memory aware', which will
>>>>> allow it to address 4 GB, but then only on operating systems that
>>>>> can address>2 GB (e.g., not 32 bit operating systems).
>>>>>
>>>>> Note that such measures will only help if the memory exhaustion is
>>>>> being caused by the amount/complexity of content. If the problem
>>>>> is truly a memory leak, this workaround will only prolong the
>>>>> inevitable, as it will simply provide more memory for the leaking.
>>>>>
>>>>> I saw this problem *very* briefly in the 0.7.3 series about a
>>>>> month ago. I never did have any indication of what was causing it,
>>>>> and it occurred day over day, without significant changes to the
>>>>> content. It went away just as fast as it came.
>>>>>
>>>>>
>>>>>
>>>>> FYI this is on a 25 region megaregion with>50k prims and at least
>>>>> several rather complex scripts in play.
>>>>>
>>>>>
>>>>> Cheers
>>>>> James/Hiro
>>>>>
>>>>>
>>>>> On Mon, Feb 13, 2012 at 8:39 PM, Gary Banham<banham1 at telus.net
>>>>> <mailto:banham1 at telus.net>> wrote:
>>>>>
>>>>> Fleep I’m running into exactly the same problem I check the
>>>>> flotsam cache to make sure I wasn’t using memory
>>>>>
>>>>> My server runs windows seven but has the same amount of memory
>>>>> I have about 8 regions on the server some with quite a few
>>>>> scripts
>>>>>
>>>>> I rolled back the instance to 7.1.1 to see what would happen
>>>>> and it worked fine there must be something different in the
>>>>> way 7.2 release uses the memory
>>>>>
>>>>> I’m in the process of distributing the regions over a number
>>>>> of 7.2 instances I’m not sure if that’s going to cause other
>>>>> problems
>>>>>
>>>>> TC
>>>>>
>>>>> Gary
>>>>>
>>>>> *From:*opensim-users-bounces at lists.berlios.de
>>>>> <mailto:opensim-users-bounces at lists.berlios.de>
>>>>> [mailto:opensim-users-bounces at lists.berlios.de
>>>>> <mailto:opensim-users-bounces at lists.berlios.de>] *On Behalf Of
>>>>> *Fleep Tuque
>>>>> *Sent:* February-13-12 11:06 AM
>>>>> *To:* opensim-users at lists.berlios.de
>>>>> <mailto:opensim-users at lists.berlios.de>
>>>>> *Subject:* [Opensim-users] Intermittent crashing -
>>>>> System.OutofMemoryException
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Our campus opensim system (running version 0.7.2 Release in
>>>>> grid mode) seems to intermittently be crashing the last week
>>>>> or so and I can't figure out why or how to reproduce it. An
>>>>> out of memory exception doesn't make sense to me, the server
>>>>> has 24GB RAM and has very light load since we're still in
>>>>> pilot/testing phase. No more than 5 or 6 users concurrent,
>>>>> only a few sims with much content..
>>>>>
>>>>> Does this error mean anything to anyone?
>>>>>
>>>>> 012-02-11 18:18:33,738 ERROR - OpenSim.Application [APPLICATION]:
>>>>>
>>>>> APPLICATION EXCEPTION DETECTED:
>>>>> System.UnhandledExceptionEventArgs
>>>>>
>>>>> Exception: System.OutOfMemoryException: Exception of type
>>>>> 'System.OutOfMemoryException' was thrown.
>>>>>
>>>>> at System.Threading.Thread.StartInternal(IPrincipal principal,
>>>>> StackCrawlMark& stackMark)
>>>>>
>>>>> at System.Threading.Thread.Start()
>>>>>
>>>>> at Amib.Threading.SmartThreadPool.StartThreads(Int32
>>>>> threadsCount)
>>>>>
>>>>> at Amib.Threading.SmartThreadPool.Enqueue(WorkItem workItem,
>>>>> Boolean incrementWorkItems)
>>>>>
>>>>> at
>>>>> Amib.Threading.SmartThreadPool.QueueWorkItem(WorkItemCallback
>>>>> callback, Object state)
>>>>>
>>>>> at OpenSim.Framework.Util.FireAndForget(WaitCallback callback,
>>>>> Object obj)
>>>>>
>>>>> at
>>>>>
>>>>> OpenSim.Region.ClientStack.LindenUDP.LLUDPServer.PacketReceived(UDPPacketBuffer
>>>>>
>>>>> buffer)
>>>>>
>>>>> at OpenMetaverse.OpenSimUDPBase.AsyncEndReceive(IAsyncResult iar)
>>>>>
>>>>> at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
>>>>>
>>>>> at System.Net.ContextAwareResult.CompleteCallback(Object state)
>>>>>
>>>>> at System.Threading.ExecutionContext.runTryCode(Object userData)
>>>>>
>>>>> at
>>>>>
>>>>> System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode
>>>>>
>>>>> code, CleanupCode backoutCode, Object userData)
>>>>>
>>>>> at
>>>>> System.Threading.ExecutionContext.RunInternal(ExecutionContext
>>>>> executionContext, ContextCallback callback, Object state)
>>>>>
>>>>> at System.Threading.ExecutionContext.Run(ExecutionContext
>>>>> executionContext, ContextCallback callback, Object state)
>>>>>
>>>>> at System.Net.ContextAwareResult.Complete(IntPtr userToken)
>>>>>
>>>>> at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object
>>>>> result, IntPtr userToken)
>>>>>
>>>>> at
>>>>>
>>>>> System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32
>>>>>
>>>>> errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
>>>>>
>>>>> at
>>>>>
>>>>> System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32
>>>>>
>>>>> errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
>>>>>
>>>>> Application is terminating: True
>>>>>
>>>>> Any pointers or advice appreciated!
>>>>>
>>>>> - Chris/Fleep
>>>>>
>>>>> Chris M. Collins (SL/OS: Fleep Tuque)
>>>>>
>>>>> Center for Simulations& Virtual Environments Research (UCSIM)
>>>>>
>>>>> UCIT Instructional& Research Computing
>>>>>
>>>>>
>>>>> University of Cincinnati
>>>>>
>>>>> 406A Zimmer Hall
>>>>>
>>>>> 315 College Drive
>>>>>
>>>>> PO BOX 210088
>>>>>
>>>>> Cincinnati, OH 45221-0088
>>>>>
>>>>> chris.collins at uc.edu<mailto:chris.collins at uc.edu>
>>>>>
>>>>> (513) 556-3018<tel:%28513%29%20556-3018>
>>>>>
>>>>> http://ucsim.uc.edu
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Opensim-users mailing list
>>>>> Opensim-users at lists.berlios.de
>>>>> <mailto:Opensim-users at lists.berlios.de>
>>>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ===================================
>>>>> http://simhost.com
>>>>> http://twitter.com/jstallings2
>>>>> http://www.linkedin.com/pub/5/770/a49
>>>>>
>>>>> _______________________________________________
>>>>> Opensim-users mailing list
>>>>> Opensim-users at lists.berlios.de<mailto:Opensim-users at lists.berlios.de>
>>>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Opensim-users mailing list
>>>>> Opensim-users at lists.berlios.de
>>>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Justin Clark-Casey (justincc)
>>> http://justincc.org/blog
>>> http://twitter.com/justincc
>>> _______________________________________________
>>> Opensim-users mailing list
>>> Opensim-users at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>
>>
>>
>
>