[Opensim-users] Intermittent crashing - System.OutofMemoryException
Justin Clark-Casey
jjustincc at googlemail.com
Wed Feb 15 02:50:21 UTC 2012
It's a good question. I'm not sure there any such conditions - it appears that the time saved by reading this out from
memory is dwarfed by other sources of delay (chiefly network, I expect). All of the 'serious' simulator operators that
I know never go near Flotsam's memory cache.
James' point about the 2MB limit is a great one - I had forgotten about this on Windows. This is the problem that
you'll hit first no matter how much memory you have (which is part of the reason why people run multiple instances of
OpenSimulator rather than a large number of regions on a single simulator).
I believe that it's perfect possible to get Windows to use 64 bit libraries when appropriate but this requires doing a
bit of grunt work (possibly compiling 64 bit versions of ode.dll and sqlite3.dll and then putting the required detection
code into OpenSim).
On 14/02/12 15:13, core wrote:
> this is an interesting and informative thread. both fleep's and gary's
> comments about flotsam cache settings have made me wonder - exactly
> under what conditions should, if at all, flotsam memory caching be set
> to TRUE. i've tried it both ways and have seen little difference, but
> admittedly this was under moderate usage at best. since there are plenty
> of people here running live-fire instances in a variety of environments
> i was hoping to gain some insight.
>
> thanks to hiro for the excellent breakdown of .net memory usage. and to
> fleep, we all know cat hair is in fact magical and makes our gear run
> faster and smarter ;)
>
> - core
>
> On 2/14/2012 7:09 AM, Fleep Tuque wrote:
>> Thanks all for the info and thoughts, sounds like scheduling daily
>> restarts is probably the best option.
>>
>> I'm pretty sure I haven't messed with whatever the default settings
>> are for the flotsam cache, I never copy over old config files when I
>> upgrade, I go through and update by hand and usually just base
>> settings to make sure the upgrade was successful and then turn on
>> other options that we've tested.
>>
>> I have to say, it's almost funny that FleepGrid, which is running
>> Windows XP on a P4 box that was gathering dust and cat hair in the
>> basement, 4GB of RAM, and at least 8 sims full of content and scripts
>> isn't having that problem, but the new poweredge server running
>> WinServer2008, tons of RAM, and only 3 or 4 sims of content is flaking
>> out. In fact, three of the regions on the UCSIM grid are identical to
>> the ones on FleepGrid since I ported them over as OAR files, so it
>> can't be much of a difference in content..
>>
>> Think I'll try going through the log files again, I'd sure like to
>> know what's tipping the out of memory exception!
>>
>> Thanks again.
>>
>> - Chris/Fleep
>>
>> Chris M. Collins (SL/OS: Fleep Tuque)
>> Center for Simulations& Virtual Environments Research (UCSIM)
>> UCIT Instructional& Research Computing
>> University of Cincinnati
>> 406A Zimmer Hall
>> 315 College Drive
>> PO BOX 210088
>> Cincinnati, OH 45221-0088
>> chris.collins at uc.edu<mailto:chris.collins at uc.edu>
>> (513) 556-3018
>>
>> http://ucsim.uc.edu
>>
>> On Tue, Feb 14, 2012 at 7:29 AM, James Stallings II
>> <james.stallings at gmail.com<mailto:james.stallings at gmail.com>> wrote:
>>
>> Regardless the amount of RAM on your system, the .net runtime uses
>> a memory model of a specific size (2 GB) and cannot address more
>> than that except under special circumstances.
>>
>> There is a hack around, involving the use of a Visual Studio C++
>> utility to make a .net binary 'large memory aware', which will
>> allow it to address 4 GB, but then only on operating systems that
>> can address>2 GB (e.g., not 32 bit operating systems).
>>
>> Note that such measures will only help if the memory exhaustion is
>> being caused by the amount/complexity of content. If the problem
>> is truly a memory leak, this workaround will only prolong the
>> inevitable, as it will simply provide more memory for the leaking.
>>
>> I saw this problem *very* briefly in the 0.7.3 series about a
>> month ago. I never did have any indication of what was causing it,
>> and it occurred day over day, without significant changes to the
>> content. It went away just as fast as it came.
>>
>>
>>
>> FYI this is on a 25 region megaregion with>50k prims and at least
>> several rather complex scripts in play.
>>
>>
>> Cheers
>> James/Hiro
>>
>>
>> On Mon, Feb 13, 2012 at 8:39 PM, Gary Banham<banham1 at telus.net
>> <mailto:banham1 at telus.net>> wrote:
>>
>> Fleep I’m running into exactly the same problem I check the
>> flotsam cache to make sure I wasn’t using memory
>>
>> My server runs windows seven but has the same amount of memory
>> I have about 8 regions on the server some with quite a few scripts
>>
>> I rolled back the instance to 7.1.1 to see what would happen
>> and it worked fine there must be something different in the
>> way 7.2 release uses the memory
>>
>> I’m in the process of distributing the regions over a number
>> of 7.2 instances I’m not sure if that’s going to cause other
>> problems
>>
>> TC
>>
>> Gary
>>
>> *From:*opensim-users-bounces at lists.berlios.de
>> <mailto:opensim-users-bounces at lists.berlios.de>
>> [mailto:opensim-users-bounces at lists.berlios.de
>> <mailto:opensim-users-bounces at lists.berlios.de>] *On Behalf Of
>> *Fleep Tuque
>> *Sent:* February-13-12 11:06 AM
>> *To:* opensim-users at lists.berlios.de
>> <mailto:opensim-users at lists.berlios.de>
>> *Subject:* [Opensim-users] Intermittent crashing -
>> System.OutofMemoryException
>>
>> Hi all,
>>
>> Our campus opensim system (running version 0.7.2 Release in
>> grid mode) seems to intermittently be crashing the last week
>> or so and I can't figure out why or how to reproduce it. An
>> out of memory exception doesn't make sense to me, the server
>> has 24GB RAM and has very light load since we're still in
>> pilot/testing phase. No more than 5 or 6 users concurrent,
>> only a few sims with much content..
>>
>> Does this error mean anything to anyone?
>>
>> 012-02-11 18:18:33,738 ERROR - OpenSim.Application [APPLICATION]:
>>
>> APPLICATION EXCEPTION DETECTED: System.UnhandledExceptionEventArgs
>>
>> Exception: System.OutOfMemoryException: Exception of type
>> 'System.OutOfMemoryException' was thrown.
>>
>> at System.Threading.Thread.StartInternal(IPrincipal principal,
>> StackCrawlMark& stackMark)
>>
>> at System.Threading.Thread.Start()
>>
>> at Amib.Threading.SmartThreadPool.StartThreads(Int32 threadsCount)
>>
>> at Amib.Threading.SmartThreadPool.Enqueue(WorkItem workItem,
>> Boolean incrementWorkItems)
>>
>> at
>> Amib.Threading.SmartThreadPool.QueueWorkItem(WorkItemCallback
>> callback, Object state)
>>
>> at OpenSim.Framework.Util.FireAndForget(WaitCallback callback,
>> Object obj)
>>
>> at
>> OpenSim.Region.ClientStack.LindenUDP.LLUDPServer.PacketReceived(UDPPacketBuffer
>> buffer)
>>
>> at OpenMetaverse.OpenSimUDPBase.AsyncEndReceive(IAsyncResult iar)
>>
>> at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
>>
>> at System.Net.ContextAwareResult.CompleteCallback(Object state)
>>
>> at System.Threading.ExecutionContext.runTryCode(Object userData)
>>
>> at
>> System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode
>> code, CleanupCode backoutCode, Object userData)
>>
>> at
>> System.Threading.ExecutionContext.RunInternal(ExecutionContext
>> executionContext, ContextCallback callback, Object state)
>>
>> at System.Threading.ExecutionContext.Run(ExecutionContext
>> executionContext, ContextCallback callback, Object state)
>>
>> at System.Net.ContextAwareResult.Complete(IntPtr userToken)
>>
>> at System.Net.LazyAsyncResult.ProtectedInvokeCallback(Object
>> result, IntPtr userToken)
>>
>> at
>> System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32
>> errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
>>
>> at
>> System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32
>> errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
>>
>> Application is terminating: True
>>
>> Any pointers or advice appreciated!
>>
>> - Chris/Fleep
>>
>> Chris M. Collins (SL/OS: Fleep Tuque)
>>
>> Center for Simulations& Virtual Environments Research (UCSIM)
>>
>> UCIT Instructional& Research Computing
>>
>> University of Cincinnati
>>
>> 406A Zimmer Hall
>>
>> 315 College Drive
>>
>> PO BOX 210088
>>
>> Cincinnati, OH 45221-0088
>>
>> chris.collins at uc.edu<mailto:chris.collins at uc.edu>
>>
>> (513) 556-3018<tel:%28513%29%20556-3018>
>>
>> http://ucsim.uc.edu
>>
>>
>> _______________________________________________
>> Opensim-users mailing list
>> Opensim-users at lists.berlios.de
>> <mailto:Opensim-users at lists.berlios.de>
>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>
>>
>>
>>
>> --
>> ===================================
>> http://simhost.com
>> http://twitter.com/jstallings2
>> http://www.linkedin.com/pub/5/770/a49
>>
>> _______________________________________________
>> Opensim-users mailing list
>> Opensim-users at lists.berlios.de<mailto:Opensim-users at lists.berlios.de>
>> https://lists.berlios.de/mailman/listinfo/opensim-users
>>
>>
>>
>>
>> _______________________________________________
>> Opensim-users mailing list
>> Opensim-users at lists.berlios.de
>> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
More information about the Opensim-users
mailing list