[Opensim-users] Intermittent crashing - System.OutofMemoryException
Rick Anderson
rianders at docs.rutgers.edu
Wed Feb 15 14:58:08 UTC 2012
Since one machine is faster than another is there a chance a process
that leaks memory is happening faster? Can the memory usage be shown
to go to the 2 gig limit then fail? What's a good way to check?
Could there be an issue with permissions/settings from OAR import?
Like ownership and groups not matching. Then the generated errors
consume memory?
A couple of random thoughts,
-_Rick
On Tue, Feb 14, 2012 at 9:50 PM, Justin Clark-Casey
<jjustincc at googlemail.com> wrote:
> It's a good question. I'm not sure there any such conditions - it appears
> that the time saved by reading this out from memory is dwarfed by other
> sources of delay (chiefly network, I expect). All of the 'serious'
> simulator operators that I know never go near Flotsam's memory cache.
>
> James' point about the 2MB limit is a great one - I had forgotten about this
> on Windows. This is the problem that you'll hit first no matter how much
> memory you have (which is part of the reason why people run multiple
> instances of OpenSimulator rather than a large number of regions on a single
> simulator).
>
> I believe that it's perfect possible to get Windows to use 64 bit libraries
> when appropriate but this requires doing a bit of grunt work (possibly
> compiling 64 bit versions of ode.dll and sqlite3.dll and then putting the
> required detection code into OpenSim).
>
>
> On 14/02/12 15:13, core wrote:
>>
>> this is an interesting and informative thread. both fleep's and gary's
>> comments about flotsam cache settings have made me wonder - exactly
>> under what conditions should, if at all, flotsam memory caching be set
>> to TRUE. i've tried it both ways and have seen little difference, but
>> admittedly this was under moderate usage at best. since there are plenty
>> of people here running live-fire instances in a variety of environments
>> i was hoping to gain some insight.
>>
>> thanks to hiro for the excellent breakdown of .net memory usage. and to
>> fleep, we all know cat hair is in fact magical and makes our gear run
>> faster and smarter ;)
>>
>> - core
>>
>> On 2/14/2012 7:09 AM, Fleep Tuque wrote:
>>>
>>> Thanks all for the info and thoughts, sounds like scheduling daily
>>> restarts is probably the best option.
>>>
>>> I'm pretty sure I haven't messed with whatever the default settings
>>> are for the flotsam cache, I never copy over old config files when I
>>> upgrade, I go through and update by hand and usually just base
>>> settings to make sure the upgrade was successful and then turn on
>>> other options that we've tested.
>>>
>>> I have to say, it's almost funny that FleepGrid, which is running
>>> Windows XP on a P4 box that was gathering dust and cat hair in the
>>> basement, 4GB of RAM, and at least 8 sims full of content and scripts
>>> isn't having that problem, but the new poweredge server running
>>> WinServer2008, tons of RAM, and only 3 or 4 sims of content is flaking
>>> out. In fact, three of the regions on the UCSIM grid are identical to
>>> the ones on FleepGrid since I ported them over as OAR files, so it
>>> can't be much of a difference in content..
>>>
>>> Think I'll try going through the log files again, I'd sure like to
>>> know what's tipping the out of memory exception!
>>>
>>> Thanks again.
>>>
>>> - Chris/Fleep
>>>
>>> Chris M. Collins (SL/OS: Fleep Tuque)
>>> Center for Simulations& Virtual Environments Research (UCSIM)
>>> UCIT Instructional& Research Computing
>>>
>>> University of Cincinnati
>>> 406A Zimmer Hall
>>> 315 College Drive
>>> PO BOX 210088
>>> Cincinnati, OH 45221-0088
>>> chris.collins at uc.edu<mailto:chris.collins at uc.edu>
>>> (513) 556-3018
>>>
>>> http://ucsim.uc.edu
>>>
>>> On Tue, Feb 14, 2012 at 7:29 AM, James Stallings II
>>> <james.stallings at gmail.com<mailto:james.stallings at gmail.com>> wrote:
>>>
>>> Regardless the amount of RAM on your system, the .net runtime uses
>>> a memory model of a specific size (2 GB) and cannot address more
>>> than that except under special circumstances.
>>>
>>> There is a hack around, involving the use of a Visual Studio C++
>>> utility to make a .net binary 'large memory aware', which will
>>> allow it to address 4 GB, but then only on operating systems that
>>> can address>2 GB (e.g., not 32 bit operating systems).
>>>
>>> Note that such measures will only help if the memory exhaustion is
>>> being caused by the amount/complexity of content. If the problem
>>> is truly a memory leak, this workaround will only prolong the
>>> inevitable, as it will simply provide more memory for the leaking.
>>>
>>> I saw this problem *very* briefly in the 0.7.3 series about a
>>> month ago. I never did have any indication of what was causing it,
>>> and it occurred day over day, without significant changes to the
>>> content. It went away just as fast as it came.
>>>
>>>
>>>
>>> FYI this is on a 25 region megaregion with>50k prims and at least
>>> several rather complex scripts in play.
>>>
>>>
>>> Cheers
>>> James/Hiro
>>>
>>>
>>> On Mon, Feb 13, 2012 at 8:39 PM, Gary Banham<banham1 at telus.net
>>> <mailto:banham1 at telus.net>> wrote:
>>>
>>> Fleep I’m running into exactly the same problem I check the
>>> flotsam cache to make sure I wasn’t using memory
>>>
>>> My server runs windows seven but has the same amount of memory
>>> I have about 8 regions on the server some with quite a few
>>> scripts
>>>
>>> I rolled back the instance to 7.1.1 to see what would happen
>>> and it worked fine there must be something different in the
>>> way 7.2 release uses the memory
>>>
>>> I’m in the process of distributing the regions over a number
>>> of 7.2 instances I’m not sure if that’s going to cause other
>>> problems
>>>
>>> TC
>>>
>>> Gary
>>>
>>> *From:*opensim-users-bounces at lists.berlios.de
>>> <mailto:opensim-users-bounces at lists.berlios.de>
>>> [mailto:opensim-users-bounces at lists.berlios.de
>>> <mailto:opensim-users-bounces at lists.berlios.de>] *On Behalf Of
>>> *Fleep Tuque
>>> *Sent:* February-13-12 11:06 AM
>>> *To:* opensim-users at lists.berlios.de
>>> <mailto:opensim-users at lists.berlios.de>
>>> *Subject:* [Opensim-users] Intermittent crashing -
>>> System.OutofMemoryException
>>>
>>> Hi all,
>>>
>>> Our campus opensim system (running version 0.7.2 Release in
>>> grid mode) seems to intermittently be crashing the last week
>>> or so and I can't figure out why or how to reproduce it. An
>>> out of memory exception doesn't make sense to me, the server
>>> has 24GB RAM and has very light load since we're still in
>>> pilot/testing phase. No more than 5 or 6 users concurrent,
>>> only a few sims with much content..
>>>
>>> Does this error mean anything to anyone?
>>>
>>> 012-02-11 18:18:33,738 ERROR - OpenSim.Application [APPLICATION]:
>>>
>>> APPLICATION EXCEPTION DETECTED:
>>> System.UnhandledExceptionEventArgs
>>>
>>> Exception: System.OutOfMemoryException: Exception of type
>>> 'System.OutOfMemoryException' was thrown.
>>>
>>> at System.Threading.Thread.StartInternal(IPrincipal principal,
>>> StackCrawlMark& stackMark)
>>>
>>> at System.Threading.Thread.Start()
>>>
>>> at Amib.Threading.SmartThreadPool.StartThreads(Int32
>>> threadsCount)
>>>
>>> at Amib.Threading.SmartThreadPool.Enqueue(WorkItem workItem,
>>> Boolean incrementWorkItems)
>>>
>>> at
>>> Amib.Threading.SmartThreadPool.QueueWorkItem(WorkItemCallback
>>> callback, Object state)
>>>
>>> at OpenSim.Framework.Util.FireAndForget(WaitCallback callback,
>>> Object obj)
>>>
>>> at
>>>
>>> OpenSim.Region.ClientStack.LindenUDP.LLUDPServer.PacketReceived(UDPPacketBuffer
>>> buffer)
>>>
>>> at OpenMetaverse.OpenSimUDPBase.AsyncEndReceive(IAsyncResult iar)
>>>
>>> at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
>>>
>>> at System.Net.ContextAwareResult.CompleteCallback(Object state)
>>>
>>> at System.Threading.ExecutionContext.runTryCode(Object userData)
>>>
>>> at
>>>
>>> System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode
>>> code, CleanupCode backoutCode, Object userData)
>>>
>>> at
>>> System.Threading.ExecutionContext.RunInternal(ExecutionContext
>>> executionContext, ContextCallback callback, Object state)
>>>
>>> at System.Threading.ExecutionContext.Run(ExecutionContext
>>> executionContext, ContextCallback callback, Object state)
>>>
>>> at System.Net.ContextAwareResult.Complete(IntPtr userToken)
>>>
>>> at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object
>>> result, IntPtr userToken)
>>>
>>> at
>>>
>>> System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32
>>> errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
>>>
>>> at
>>>
>>> System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32
>>> errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
>>>
>>> Application is terminating: True
>>>
>>> Any pointers or advice appreciated!
>>>
>>> - Chris/Fleep
>>>
>>> Chris M. Collins (SL/OS: Fleep Tuque)
>>>
>>> Center for Simulations& Virtual Environments Research (UCSIM)
>>>
>>> UCIT Instructional& Research Computing
>>>
>>>
>>> University of Cincinnati
>>>
>>> 406A Zimmer Hall
>>>
>>> 315 College Drive
>>>
>>> PO BOX 210088
>>>
>>> Cincinnati, OH 45221-0088
>>>
>>> chris.collins at uc.edu<mailto:chris.collins at uc.edu>
>>>
>>> (513) 556-3018<tel:%28513%29%20556-3018>
>>>
>>> http://ucsim.uc.edu
>>>
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> Opensim-users at lists.berlios.de
>>> <mailto:Opensim-users at lists.berlios.de>
>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>
>>>
>>>
>>>
>>> --
>>> ===================================
>>> http://simhost.com
>>> http://twitter.com/jstallings2
>>> http://www.linkedin.com/pub/5/770/a49
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> Opensim-users at lists.berlios.de<mailto:Opensim-users at lists.berlios.de>
>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Opensim-users mailing list
>>> Opensim-users at lists.berlios.de
>>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>
>>
>>
>
>
> --
> Justin Clark-Casey (justincc)
> http://justincc.org/blog
> http://twitter.com/justincc
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-users
--
Rick Anderson
Director of Virtual Worlds
Division of Continuing Studies (DoCS)
Rutgers University
(732) 586-3265
More information about the Opensim-users
mailing list