[Opensim-users] Intermittent crashing - System.OutofMemoryException

BlueWall jamesh at bluewallgroup.com
Thu Feb 16 15:49:06 UTC 2012 

The rrdtools is very good for storing and organizing the data for 
statistics and it will also produce graphs from that data. It relies on 
external scripts to collect the data - that's where Cacti comes in. It 
has the scripts and templates to graph your system (and snmp devices) in 
the distribution. I have done it on Linux and they claim to work on 
Windows too.

You could also write scripts and templates to graph OpenSim. I haven't 
tried that, though. There is documentation included in the distro, and 
scattered across the Cacti user forums.

-BlueWall

On 02/16/2012 09:55 AM, Rick Anderson wrote:
> One question about Cacti, and RDDTools. The sites describe them as
> general purpose graphing tools. Is there any documentation or tutorial
> for configuring them to monitor system status. Maybe, it's really
> simple, and I'm over complicating it.
>
> -_Rick
>
> On Thu, Feb 16, 2012 at 6:53 AM, BlueWall<jamesh at bluewallgroup.com>  wrote:
>>
>> Cacti will work -
>> http://www.cacti.net/downloads/docs/html/install_windows.html
>>
>>
>> On 02/15/2012 03:24 PM, Justin Clark-Casey wrote:
>>>
>>> Generated errors will not consume memory.
>>>
>>> The best way to track usage is to use a system monitoring tool like
>>> rrdtool to record process memory use (not sure what the Windows
>>> equivalent is). You can also extract statistics from OpenSim itself [1]
>>> and record them in the same tool.
>>>
>>> I'm not sure if there is any public code on how to glue these things
>>> together, though I remember that osgrid was recording stats at some point.
>>>
>>> If one can record stats then you can see memory usage and starting
>>> twiddling some config things to try and pin it down. For instance, I
>>> have reason to be believe that right now, running a large number of
>>> scripts might leak memory even if no avatars are present in the region.
>>> I don't know the cause of this, whether it's a general script issue or
>>> some particular LSL operation.
>>>
>>> Unfortunately, this is getting into fairly complex diagnostics but
>>> running a high uptime simulator with significant usage is not easy. The
>>> alternative solution is simply to restart the simulator periodically
>>> (e.g. daily) in order to reset the memory.
>>>
>>> [1] http://opensimulator.org/wiki/Monitoring
>>>
>>> On 15/02/12 14:58, Rick Anderson wrote:
>>>>
>>>> Since one machine is faster than another is there a chance a process
>>>> that leaks memory is happening faster? Can the memory usage be shown
>>>> to go to the 2 gig limit then fail? What's a good way to check?
>>>>
>>>> Could there be an issue with permissions/settings from OAR import?
>>>> Like ownership and groups not matching. Then the generated errors
>>>> consume memory?
>>>>
>>>> A couple of random thoughts,
>>>> -_Rick
>>>>
>>>> On Tue, Feb 14, 2012 at 9:50 PM, Justin Clark-Casey
>>>> <jjustincc at googlemail.com>  wrote:
>>>>>
>>>>> It's a good question. I'm not sure there any such conditions - it
>>>>> appears
>>>>> that the time saved by reading this out from memory is dwarfed by other
>>>>> sources of delay (chiefly network, I expect). All of the 'serious'
>>>>> simulator operators that I know never go near Flotsam's memory cache.
>>>>>
>>>>> James' point about the 2MB limit is a great one - I had forgotten
>>>>> about this
>>>>> on Windows. This is the problem that you'll hit first no matter how much
>>>>> memory you have (which is part of the reason why people run multiple
>>>>> instances of OpenSimulator rather than a large number of regions on a
>>>>> single
>>>>> simulator).
>>>>>
>>>>> I believe that it's perfect possible to get Windows to use 64 bit
>>>>> libraries
>>>>> when appropriate but this requires doing a bit of grunt work (possibly
>>>>> compiling 64 bit versions of ode.dll and sqlite3.dll and then putting
>>>>> the
>>>>> required detection code into OpenSim).
>>>>>
>>>>>
>>>>> On 14/02/12 15:13, core wrote:
>>>>>>
>>>>>>
>>>>>> this is an interesting and informative thread. both fleep's and gary's
>>>>>> comments about flotsam cache settings have made me wonder - exactly
>>>>>> under what conditions should, if at all, flotsam memory caching be set
>>>>>> to TRUE. i've tried it both ways and have seen little difference, but
>>>>>> admittedly this was under moderate usage at best. since there are
>>>>>> plenty
>>>>>> of people here running live-fire instances in a variety of environments
>>>>>> i was hoping to gain some insight.
>>>>>>
>>>>>> thanks to hiro for the excellent breakdown of .net memory usage. and to
>>>>>> fleep, we all know cat hair is in fact magical and makes our gear run
>>>>>> faster and smarter ;)
>>>>>>
>>>>>> - core
>>>>>>
>>>>>> On 2/14/2012 7:09 AM, Fleep Tuque wrote:
>>>>>>>
>>>>>>>
>>>>>>> Thanks all for the info and thoughts, sounds like scheduling daily
>>>>>>> restarts is probably the best option.
>>>>>>>
>>>>>>> I'm pretty sure I haven't messed with whatever the default settings
>>>>>>> are for the flotsam cache, I never copy over old config files when I
>>>>>>> upgrade, I go through and update by hand and usually just base
>>>>>>> settings to make sure the upgrade was successful and then turn on
>>>>>>> other options that we've tested.
>>>>>>>
>>>>>>> I have to say, it's almost funny that FleepGrid, which is running
>>>>>>> Windows XP on a P4 box that was gathering dust and cat hair in the
>>>>>>> basement, 4GB of RAM, and at least 8 sims full of content and scripts
>>>>>>> isn't having that problem, but the new poweredge server running
>>>>>>> WinServer2008, tons of RAM, and only 3 or 4 sims of content is flaking
>>>>>>> out. In fact, three of the regions on the UCSIM grid are identical to
>>>>>>> the ones on FleepGrid since I ported them over as OAR files, so it
>>>>>>> can't be much of a difference in content..
>>>>>>>
>>>>>>> Think I'll try going through the log files again, I'd sure like to
>>>>>>> know what's tipping the out of memory exception!
>>>>>>>
>>>>>>> Thanks again.
>>>>>>>
>>>>>>> - Chris/Fleep
>>>>>>>
>>>>>>> Chris M. Collins (SL/OS: Fleep Tuque)
>>>>>>> Center for Simulations&  Virtual Environments Research (UCSIM)
>>>>>>> UCIT Instructional&  Research Computing
>>>>>>>
>>>>>>> University of Cincinnati
>>>>>>> 406A Zimmer Hall
>>>>>>> 315 College Drive
>>>>>>> PO BOX 210088
>>>>>>> Cincinnati, OH 45221-0088
>>>>>>> chris.collins at uc.edu<mailto:chris.collins at uc.edu>
>>>>>>> (513) 556-3018
>>>>>>>
>>>>>>> http://ucsim.uc.edu
>>>>>>>
>>>>>>> On Tue, Feb 14, 2012 at 7:29 AM, James Stallings II
>>>>>>> <james.stallings at gmail.com<mailto:james.stallings at gmail.com>>  wrote:
>>>>>>>
>>>>>>> Regardless the amount of RAM on your system, the .net runtime uses
>>>>>>> a memory model of a specific size (2 GB) and cannot address more
>>>>>>> than that except under special circumstances.
>>>>>>>
>>>>>>> There is a hack around, involving the use of a Visual Studio C++
>>>>>>> utility to make a .net binary 'large memory aware', which will
>>>>>>> allow it to address 4 GB, but then only on operating systems that
>>>>>>> can address>2 GB (e.g., not 32 bit operating systems).
>>>>>>>
>>>>>>> Note that such measures will only help if the memory exhaustion is
>>>>>>> being caused by the amount/complexity of content. If the problem
>>>>>>> is truly a memory leak, this workaround will only prolong the
>>>>>>> inevitable, as it will simply provide more memory for the leaking.
>>>>>>>
>>>>>>> I saw this problem *very* briefly in the 0.7.3 series about a
>>>>>>> month ago. I never did have any indication of what was causing it,
>>>>>>> and it occurred day over day, without significant changes to the
>>>>>>> content. It went away just as fast as it came.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> FYI this is on a 25 region megaregion with>50k prims and at least
>>>>>>> several rather complex scripts in play.
>>>>>>>
>>>>>>>
>>>>>>> Cheers
>>>>>>> James/Hiro
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 13, 2012 at 8:39 PM, Gary Banham<banham1 at telus.net
>>>>>>> <mailto:banham1 at telus.net>>  wrote:
>>>>>>>
>>>>>>> Fleep I’m running into exactly the same problem I check the
>>>>>>> flotsam cache to make sure I wasn’t using memory
>>>>>>>
>>>>>>> My server runs windows seven but has the same amount of memory
>>>>>>> I have about 8 regions on the server some with quite a few
>>>>>>> scripts
>>>>>>>
>>>>>>> I rolled back the instance to 7.1.1 to see what would happen
>>>>>>> and it worked fine there must be something different in the
>>>>>>> way 7.2 release uses the memory
>>>>>>>
>>>>>>> I’m in the process of distributing the regions over a number
>>>>>>> of 7.2 instances I’m not sure if that’s going to cause other
>>>>>>> problems
>>>>>>>
>>>>>>> TC
>>>>>>>
>>>>>>> Gary
>>>>>>>
>>>>>>> *From:*opensim-users-bounces at lists.berlios.de
>>>>>>> <mailto:opensim-users-bounces at lists.berlios.de>
>>>>>>> [mailto:opensim-users-bounces at lists.berlios.de
>>>>>>> <mailto:opensim-users-bounces at lists.berlios.de>] *On Behalf Of
>>>>>>> *Fleep Tuque
>>>>>>> *Sent:* February-13-12 11:06 AM
>>>>>>> *To:* opensim-users at lists.berlios.de
>>>>>>> <mailto:opensim-users at lists.berlios.de>
>>>>>>> *Subject:* [Opensim-users] Intermittent crashing -
>>>>>>> System.OutofMemoryException
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Our campus opensim system (running version 0.7.2 Release in
>>>>>>> grid mode) seems to intermittently be crashing the last week
>>>>>>> or so and I can't figure out why or how to reproduce it. An
>>>>>>> out of memory exception doesn't make sense to me, the server
>>>>>>> has 24GB RAM and has very light load since we're still in
>>>>>>> pilot/testing phase. No more than 5 or 6 users concurrent,
>>>>>>> only a few sims with much content..
>>>>>>>
>>>>>>> Does this error mean anything to anyone?
>>>>>>>
>>>>>>> 012-02-11 18:18:33,738 ERROR - OpenSim.Application [APPLICATION]:
>>>>>>>
>>>>>>> APPLICATION EXCEPTION DETECTED:
>>>>>>> System.UnhandledExceptionEventArgs
>>>>>>>
>>>>>>> Exception: System.OutOfMemoryException: Exception of type
>>>>>>> 'System.OutOfMemoryException' was thrown.
>>>>>>>
>>>>>>> at System.Threading.Thread.StartInternal(IPrincipal principal,
>>>>>>> StackCrawlMark&  stackMark)
>>>>>>>
>>>>>>> at System.Threading.Thread.Start()
>>>>>>>
>>>>>>> at Amib.Threading.SmartThreadPool.StartThreads(Int32
>>>>>>> threadsCount)
>>>>>>>
>>>>>>> at Amib.Threading.SmartThreadPool.Enqueue(WorkItem workItem,
>>>>>>> Boolean incrementWorkItems)
>>>>>>>
>>>>>>> at
>>>>>>> Amib.Threading.SmartThreadPool.QueueWorkItem(WorkItemCallback
>>>>>>> callback, Object state)
>>>>>>>
>>>>>>> at OpenSim.Framework.Util.FireAndForget(WaitCallback callback,
>>>>>>> Object obj)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> OpenSim.Region.ClientStack.LindenUDP.LLUDPServer.PacketReceived(UDPPacketBuffer
>>>>>>>
>>>>>>> buffer)
>>>>>>>
>>>>>>> at OpenMetaverse.OpenSimUDPBase.AsyncEndReceive(IAsyncResult iar)
>>>>>>>
>>>>>>> at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
>>>>>>>
>>>>>>> at System.Net.ContextAwareResult.CompleteCallback(Object state)
>>>>>>>
>>>>>>> at System.Threading.ExecutionContext.runTryCode(Object userData)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode
>>>>>>>
>>>>>>> code, CleanupCode backoutCode, Object userData)
>>>>>>>
>>>>>>> at
>>>>>>> System.Threading.ExecutionContext.RunInternal(ExecutionContext
>>>>>>> executionContext, ContextCallback callback, Object state)
>>>>>>>
>>>>>>> at System.Threading.ExecutionContext.Run(ExecutionContext
>>>>>>> executionContext, ContextCallback callback, Object state)
>>>>>>>
>>>>>>> at System.Net.ContextAwareResult.Complete(IntPtr userToken)
>>>>>>>
>>>>>>> at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object
>>>>>>> result, IntPtr userToken)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32
>>>>>>>
>>>>>>> errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>>>>
>>>>>>> System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32
>>>>>>>
>>>>>>> errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
>>>>>>>
>>>>>>> Application is terminating: True
>>>>>>>
>>>>>>> Any pointers or advice appreciated!
>>>>>>>
>>>>>>> - Chris/Fleep
>>>>>>>
>>>>>>> Chris M. Collins (SL/OS: Fleep Tuque)
>>>>>>>
>>>>>>> Center for Simulations&  Virtual Environments Research (UCSIM)
>>>>>>>
>>>>>>> UCIT Instructional&  Research Computing
>>>>>>>
>>>>>>>
>>>>>>> University of Cincinnati
>>>>>>>
>>>>>>> 406A Zimmer Hall
>>>>>>>
>>>>>>> 315 College Drive
>>>>>>>
>>>>>>> PO BOX 210088
>>>>>>>
>>>>>>> Cincinnati, OH 45221-0088
>>>>>>>
>>>>>>> chris.collins at uc.edu<mailto:chris.collins at uc.edu>
>>>>>>>
>>>>>>> (513) 556-3018<tel:%28513%29%20556-3018>
>>>>>>>
>>>>>>> http://ucsim.uc.edu
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Opensim-users mailing list
>>>>>>> Opensim-users at lists.berlios.de
>>>>>>> <mailto:Opensim-users at lists.berlios.de>
>>>>>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ===================================
>>>>>>> http://simhost.com
>>>>>>> http://twitter.com/jstallings2
>>>>>>> http://www.linkedin.com/pub/5/770/a49
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Opensim-users mailing list
>>>>>>> Opensim-users at lists.berlios.de<mailto:Opensim-users at lists.berlios.de>
>>>>>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Opensim-users mailing list
>>>>>>> Opensim-users at lists.berlios.de
>>>>>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Justin Clark-Casey (justincc)
>>>>> http://justincc.org/blog
>>>>> http://twitter.com/justincc
>>>>> _______________________________________________
>>>>> Opensim-users mailing list
>>>>> Opensim-users at lists.berlios.de
>>>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>> _______________________________________________
>> Opensim-users mailing list
>> Opensim-users at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
>

More information about the Opensim-users mailing list