[Opensim-users] Intermittent crashing - System.OutofMemoryException
Justin Clark-Casey
jjustincc at googlemail.com
Tue Feb 21 04:28:46 UTC 2012
In theory, running each simulator in it's own process should have a higher memory overhead but some aspects of
performance may be better. For instance, with ODE only one thread can call the collider at a time in right now in any
particular OpenSim process. How much this impacts performance I can't yet say but not enforcing this gets you random
hard ODE crashes.
It also just so happens that as of half an hour ago, in git master a27e5a9, OpenSim.exe can be run as a 64-bit Windows
process (I got round to getting the necessary 64-bit native Windows libraries and finding out how to get Windows to
automatically load the correct ones).
So you should be able to run OpenSim.exe now (rather than OpenSim.32BitLaunch.exe) and access upto 8TB. This is still
experimental but if it works for most people then I'll almost certainly pop it into the forthcoming 0.7.3.
On 20/02/12 12:43, Fleep Tuque wrote:
> I may be crazy, but I swear I'm seeing an immediate and very noticeable improvement in performance from the end user
> perspective. Avatar movement not as sluggish, almost snappy even, textures rezzing more quickly..
>
> The CPU usage is about the same as before the split (57-59% on average with no activity) but the Page File usage is
> higher. I'll be interested to see how things go when the HGAC visits tomorrow to give a good test with higher concurrency.
>
> Thanks again!
>
> - Chris/Fleep
>
>
> Chris M. Collins (SL/OS: Fleep Tuque)
> Center for Simulations & Virtual Environments Research (UCSIM)
> UCIT Instructional & Research Computing
> University of Cincinnati
> 406A Zimmer Hall
> 315 College Drive
> PO BOX 210088
> Cincinnati, OH 45221-0088
> chris.collins at uc.edu <mailto:chris.collins at uc.edu>
> (513) 556-3018
>
> http://ucsim.uc.edu
>
>
>
> On Mon, Feb 20, 2012 at 7:28 AM, Gary Banham <banham1 at telus.net <mailto:banham1 at telus.net>> wrote:
>
> Your more than welcome____
>
> TC____
>
> Gary____
>
> __ __
>
> *From:*opensim-users-bounces at lists.berlios.de <mailto:opensim-users-bounces at lists.berlios.de>
> [mailto:opensim-users-bounces at lists.berlios.de <mailto:opensim-users-bounces at lists.berlios.de>] *On Behalf Of *Fleep
> Tuque
> *Sent:* Monday, February 20, 2012 5:26 AM
>
>
> *To:* opensim-users at lists.berlios.de <mailto:opensim-users at lists.berlios.de>
> *Subject:* Re: [Opensim-users] Intermittent crashing - System.OutofMemoryException____
>
> __ __
>
> Wow, it really was that simple. I successfully split off a few regions into a separate instance on FleepGrid with
> no trouble at all. ____
>
> __ __
>
> The key, as Gary mentioned, was uncommenting the http_listener_port line (267 in my opensim.ini file) and
> incrementing the TCP port (in my case to 9001) for the new instance.____
>
> __ __
>
> Thanks Gary! I'll have to work out how to split the regions up logically for our UCSIM grid but I'm surprised it's
> that simple! Yay! :)____
>
> __ __
>
> - Chris/Fleep____
>
> __ __
>
> __ __
>
> Chris M. Collins (SL/OS: Fleep Tuque)____
>
> Center for Simulations & Virtual Environments Research (UCSIM)____
>
> UCIT Instructional & Research Computing____
>
> University of Cincinnati____
>
> 406A Zimmer Hall____
>
> 315 College Drive____
>
> PO BOX 210088____
>
> Cincinnati, OH 45221-0088____
>
> chris.collins at uc.edu <mailto:chris.collins at uc.edu>____
>
> (513) 556-3018 <tel:%28513%29%20556-3018>____
>
> __ __
>
> http://ucsim.uc.edu____
>
> On Mon, Feb 20, 2012 at 6:48 AM, Gary Banham <banham1 at telus.net <mailto:banham1 at telus.net>> wrote:____
>
> PS____
>
> just an additional note I think the problem is the single instance of Opensim will only access 2 gigs of memory ____
>
> with a large number of regions and lots of building and scripts the single instance can't count that high it hits a
> wall____
>
> by placing regions over a number of instances each can use 2 gigs____
>
> so yes you use more memory EG 5 instances could potentially use 10 gigs of memory but one instance only address 2
> Gigs____
>
> ____
>
> TC____
>
> Gary____
>
> ____
>
> *From:*opensim-users-bounces at lists.berlios.de <mailto:opensim-users-bounces at lists.berlios.de>
> [mailto:opensim-users-bounces at lists.berlios.de <mailto:opensim-users-bounces at lists.berlios.de>] *On Behalf Of *Fleep
> Tuque____
>
>
> *Sent:* Monday, February 20, 2012 4:34 AM
> *To:* opensim-users at lists.berlios.de <mailto:opensim-users at lists.berlios.de>____
>
> *Subject:* Re: [Opensim-users] Intermittent crashing - System.OutofMemoryException____
>
> ____
>
> I'm worried that you may be right about that and it's just a matter of time before adding more content becomes a
> problem, in which case it sounds like perhaps instead of running all the regions on a single opensim instance I
> should split them up across several as Gary mentioned above?____
>
> ____
>
> I'm not sure exactly how that works, I guess, and I'm trying to understand how that would use less memory instead of
> more.. Would I simply copy the contents of the opensim directory to multiple locations and break the regions up in
> the regions.ini file or..? ____
>
> ____
>
> I'd like to experiment with it on FleepGrid first before I hose our campus install, but the HGAC group is coming for
> a visit in a couple days so if anyone has pointers about how to do that splitting up process before I break
> anything, I'm all ears. :)____
>
> ____
>
> - Chris/Fleep____
>
> ____
>
> ____
>
> Chris M. Collins (SL/OS: Fleep Tuque)____
>
> Center for Simulations & Virtual Environments Research (UCSIM)____
>
> UCIT Instructional & Research Computing____
>
> University of Cincinnati____
>
> 406A Zimmer Hall____
>
> 315 College Drive____
>
> PO BOX 210088____
>
> Cincinnati, OH 45221-0088____
>
> chris.collins at uc.edu <mailto:chris.collins at uc.edu>____
>
> (513) 556-3018 <tel:%28513%29%20556-3018>____
>
> ____
>
> http://ucsim.uc.edu____
>
> ____
>
> ____
>
> On Thu, Feb 16, 2012 at 10:19 PM, Justin Clark-Casey <jjustincc at googlemail.com <mailto:jjustincc at googlemail.com>>
> wrote:____
>
> I would actually be rather surprised if errors during the OAR import were related to your memory problems. I think
> that it's more likely that the content of the OAR in some way pushed you beyond memory boundaries and wiping it has
> reset things.
>
> Still, if the problem is solved then that's what matters :)____
>
>
>
> On 16/02/12 22:50, Fleep Tuque wrote:____
>
> Hi all,
>
> After going through the log files and trying to figure out what changed in the time period when the out of
> memory errors
> began, we traced it back to an OAR file import. (Thanks to whoever suggested that might be an issue earlier in this
> thread!)
>
> I'd ported over a copy of the FleepGrid Plaza region so my students could easily grab some of the freebies I'd
> created,
> and it looks like something got horked during the import to the UCSIM grid:
>
> 012-02-07 09:12:05,991 ERROR - Flotsam.RegionModules.AssetCache.FlotsamAssetCache [FLOTSAM ASSET CACHE]: Failed to
> update cache for asset 3d0b3a52-40e5-48fe-8595-b72e163917fa. Exception The process cannot access the file
> 'c:\Opensim\bin\assetcache\3d0\3d0b3a52-40e5-48fe-8595-b72e163917fa' because it is being used by another
> process. at
> System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
> at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights,
> FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean
> bFromProxy)
> at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
> at System.IO.File.OpenFile(String path, FileAccess access, SafeFileHandle& handle)
> at System.IO.File.SetLastAccessTimeUtc(String path, DateTime lastAccessTimeUtc)
> at Flotsam.RegionModules.AssetCache.FlotsamAssetCache.UpdateFileCache(String key, AssetBase asset)
>
> Shortly after that, we begin to see the first out of memory errors..
>
> 2012-02-07 09:15:17,453 DEBUG - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine] Loaded script LK-Stool-Sit
> #2.poseball-zero-lag, script UUID e73e1114-297a-43a7-b4b8-dd423cd83b95, prim UUID
> 2ae59628-943d-49ec-9673-ee3465bd0242 @
> <168.1464, 156.7953, 26.90887>.UCSIM Staff 1
> 2012-02-07 09:15:17,984 ERROR - OpenSim.Region.ScriptEngine.XEngine.XEngine [XEngine] Exception creating app domain:
> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
> at System.AppDomain.nCreateDomain(String friendlyName, AppDomainSetup setup, Evidence providedSecurityInfo,
> Evidence
> creatorsSecurityInfo, IntPtr parentSecurityDescriptor)
> at System.AppDomain.CreateDomain(String friendlyName, Evidence securityInfo, AppDomainSetup info)
> at OpenSim.Region.ScriptEngine.XEngine.XEngine.DoOnRezScript(Object[] parms)
>
> And we see those out of memory errors stack up until eventually the application crashed..
>
> When we realized it was probably the OAR file that was the source of the issue, I wiped the region by importing
> a fresh
> "blank" OAR and we haven't had the out of memory exception since.
>
> Looks like the problem is resolved and super thanks again to everyone who helped with pointers and info in this
> thread -
> much appreciated!
>
> - Chris/Fleep
>
>
> Chris M. Collins (SL/OS: Fleep Tuque)
> Center for Simulations & Virtual Environments Research (UCSIM)
> UCIT Instructional & Research Computing
> University of Cincinnati
> 406A Zimmer Hall
> 315 College Drive
> PO BOX 210088
> Cincinnati, OH 45221-0088____
>
> chris.collins at uc.edu <mailto:chris.collins at uc.edu> <mailto:chris.collins at uc.edu <mailto:chris.collins at uc.edu>>
> (513) 556-3018 <tel:%28513%29%20556-3018>
>
> http://ucsim.uc.edu____
>
>
>
>
> On Thu, Feb 16, 2012 at 2:16 PM, R.Gunther <rigun at rigutech.nl <mailto:rigun at rigutech.nl>
> <mailto:rigun at rigutech.nl <mailto:rigun at rigutech.nl>>> wrote:
>
> There's a 3e option. Munin.
> http://munin-monitoring.org/
>
>
> On 2012-02-16 16:49, BlueWall wrote:
>
> The rrdtools is very good for storing and organizing the data for statistics and it will also produce graphs
> from that data. It relies on external scripts to collect the data - that's where Cacti comes in. It has the
> scripts and templates to graph your system (and snmp devices) in the distribution. I have done it on
> Linux and
> they claim to work on Windows too.
>
> You could also write scripts and templates to graph OpenSim. I haven't tried that, though. There is
> documentation included in the distro, and scattered across the Cacti user forums.
>
> -BlueWall
>
> On 02/16/2012 09:55 AM, Rick Anderson wrote:
>
> One question about Cacti, and RDDTools. The sites describe them as
> general purpose graphing tools. Is there any documentation or tutorial
> for configuring them to monitor system status. Maybe, it's really
> simple, and I'm over complicating it.
>
> -_Rick____
>
> _____________________________________________________
>
>
> Opensim-users mailing list
> Opensim-users at lists.berlios.de <mailto:Opensim-users at lists.berlios.de> <mailto:Opensim-users at lists.berlios.de
> <mailto:Opensim-users at lists.berlios.de>>____
>
> https://lists.berlios.de/__mailman/listinfo/opensim-users
> <https://lists.berlios.de/mailman/listinfo/opensim-users>____
>
>
>
>
>
>
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de <mailto:Opensim-users at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-users____
>
> ____
>
>
> --
> Justin Clark-Casey (justincc)
> http://justincc.org/blog
> http://twitter.com/justincc____
>
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de <mailto:Opensim-users at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-users____
>
> ____
>
>
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de <mailto:Opensim-users at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-users____
>
> __ __
>
>
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de <mailto:Opensim-users at lists.berlios.de>
> https://lists.berlios.de/mailman/listinfo/opensim-users
>
>
>
>
> _______________________________________________
> Opensim-users mailing list
> Opensim-users at lists.berlios.de
> https://lists.berlios.de/mailman/listinfo/opensim-users
--
Justin Clark-Casey (justincc)
http://justincc.org/blog
http://twitter.com/justincc
More information about the Opensim-users
mailing list