Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0007181opensim[REGION] Physics Enginespublic2014-05-18 15:072014-08-20 00:21
ReporterJeffKelley 
Assigned ToRobert Adams 
PrioritynormalSeveritycrashReproducibilityhave not tried
StatusassignedResolutionopen 
PlatformOSOS Version
Product Version 
Target VersionFixed in Version 
Summary0007181: BulletSPlugin : The requested operation caused a stack overflow
DescriptionThe simulator quits. No avatar inside.


2014-05-18 23:06:33,993 ERROR - OpenSim.Application [APPLICATION]:

APPLICATION EXCEPTION DETECTED: System.UnhandledExceptionEventArgs



Exception: System.StackOverflowException: The requested operation caused a stack overflow.
  at (wrapper managed-to-native) OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman/BSAPICPP:PhysicsStep2 (intptr,single,int,single,int&,int&)
  at OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman.PhysicsStep (OpenSim.Region.Physics.BulletSPlugin.BulletWorld world, Single timeStep, Int32 maxSubSteps, Single fixedTimeStep, System.Int32& updatedEntityCount, System.Int32& collidersCount) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Physics.BulletSPlugin.BSScene.DoPhysicsStep (Single timeStep) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Physics.BulletSPlugin.BSScene.Simulate (Single timeStep) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.SceneGraph.UpdatePhysics (Double elapsed) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.Scene.Update (Int32 frames) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.Scene.Heartbeat () [0x00000] in <filename unknown>:0
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0



Application is terminating: True


2014-05-18 23:06:34,025 ERROR - OpenSim.Application [APPLICATION]:

APPLICATION EXCEPTION DETECTED: System.UnhandledExceptionEventArgs



Exception: System.StackOverflowException: The requested operation caused a stack overflow.
  at (wrapper managed-to-native) OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman/BSAPICPP:PhysicsStep2 (intptr,single,int,single,int&,int&)
  at OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman.PhysicsStep (OpenSim.Region.Physics.BulletSPlugin.BulletWorld world, Single timeStep, Int32 maxSubSteps, Single fixedTimeStep, System.Int32& updatedEntityCount, System.Int32& collidersCount) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Physics.BulletSPlugin.BSScene.DoPhysicsStep (Single timeStep) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Physics.BulletSPlugin.BSScene.Simulate (Single timeStep) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.SceneGraph.UpdatePhysics (Double elapsed) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.Scene.Update (Int32 frames) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.Scene.Heartbeat () [0x00000] in <filename unknown>:0
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0



Application is terminating: True


2014-05-18 23:06:34,071 ERROR - OpenSim.Application [APPLICATION]:

APPLICATION EXCEPTION DETECTED: System.UnhandledExceptionEventArgs



Exception: System.StackOverflowException: The requested operation caused a stack overflow.
  at (wrapper managed-to-native) OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman/BSAPICPP:PhysicsStep2 (intptr,single,int,single,int&,int&)
  at OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman.PhysicsStep (OpenSim.Region.Physics.BulletSPlugin.BulletWorld world, Single timeStep, Int32 maxSubSteps, Single fixedTimeStep, System.Int32& updatedEntityCount, System.Int32& collidersCount) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Physics.BulletSPlugin.BSScene.DoPhysicsStep (Single timeStep) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Physics.BulletSPlugin.BSScene.Simulate (Single timeStep) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.SceneGraph.UpdatePhysics (Double elapsed) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.Scene.Update (Int32 frames) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.Scene.Heartbeat () [0x00000] in <filename unknown>:0
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0



Application is terminating: True
TagsNo tags attached.
Git Revision or version numberopensim-0.8-rc1
Run Mode Standalone (Multiple Regions)
Physics EngineBulletSim
Script Engine
EnvironmentMono / OSX
Mono Version3.2
Viewernone
Attached Filestxt file icon show-scene.txt [^] (4,456 bytes) 2014-05-19 16:07 [Show Content]
? file icon mono-sgen_2014-06-18-095629_u2.crash [^] (2,062,117 bytes) 2014-06-19 23:52
? file icon mono_2014-06-20-083901_u2.crash [^] (427,621 bytes) 2014-06-19 23:57
? file icon mono-sgen_2014-06-20-092700_u2.crash [^] (383,594 bytes) 2014-06-20 00:34

- Relationships

-  Notes
(0026129)
justincc (administrator)
2014-05-19 15:25

How long had the simulator been running? Is this a particularly complex scene? Are there any scripts running that are manipulating physical objects?

Output of "show scene" console command would be useful.
(0026130)
JeffKelley (reporter)
2014-05-19 16:09

The simulator has been running for about 2 hours. It usually runs 24/7 but I had to restart it after llCastRay raised hudreds of "out of memory" errors. The scene is simple, there is no physical object. Show scene attached.
(0026217)
Gavin Hird (reporter)
2014-06-01 00:40

This is the exact same behavior I have reported in http://opensimulator.org/mantis/view.php?id=6789 [^]

In my case the crash happens about 30-40 minutes after startup for both a standalone instance with 28 sims, lots of mesh and prims, but also on the public version of the same connected to OSGrid . These 28 sims has both less mesh and significantly less prims. It does not seem to have much correlation with the amount of mesh or prims.

Both sims have scripts running, the standalone significantly more (>700 more).
(0026298)
Gavin Hird (reporter)
2014-06-18 00:49

Update for version Sgrid 0.8.1 (Dev) 5450b1b: 2014-06-17 (interface version 7)

With bulletsim set as the physics engine, the mono-sgen process allocates memory at a rate of about 1 Mb/sec till it has reached max memory allocation (32-bit) and it will crash with the following error message and stack trace.

It seems to crash in OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman/BSAPICPP.PhysicsStep2




Region (root) # mono(82346,0xb6257000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(82346,0xb6257000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(82346,0xb5741000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(82346,0xb5741000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(82346,0xb9ecf000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(82346,0xb9ecf000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(82346,0xb8195000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(82346,0xb8195000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: libc++abi.dylib: libc++abi.dylib: libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_allocterminating with uncaught exception of type std::bad_alloc: std::bad_allocterminating with uncaught exception of type std::bad_alloc: std::bad_allocterminating with uncaught exception of type std::bad_alloc: std::bad_alloc



Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman/BSAPICPP.PhysicsStep2 (intptr,single,int,single,int&,int&) <0xffffffff>
mono(82346,0xb5741000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
  at OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman.PhysicsStep (OpenSim.Region.Physics.BulletSPlugin.BulletWorld,single,int,single,int&,int&) <0x00053>
  at OpenSim.Region.Physics.BulletSPlugin.BSScene.DoPhysicsStep (single) <0x002b1>
  at OpenSim.Region.Physics.BulletSPlugin.BSScene.Simulate (single) <0x00027>
  at OpenSim.Region.Framework.Scenes.SceneGraph.UpdatePhysics (double) <0x0002e>
  at OpenSim.Region.Framework.Scenes.Scene.Update (int) <0x0069f>
  at OpenSim.Region.Framework.Scenes.Scene.Heartbeat () <0x0010f>
  at System.Threading.Thread.StartInternal () <0x00057>
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

mono(82346,0xb6a67000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(82346,0xb6a67000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
mono(82346,0xb6a67000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6
(0026299)
justincc (administrator)
2014-06-18 13:38

I presume that you don't see a similar failure when using the ODE physics engine?
(0026304)
Gavin Hird (reporter)
2014-06-18 22:26

Correct, is is running nice and stable with ODE enabled.
(0026309)
justincc (administrator)
2014-06-19 15:33

Are you able to make an OAR available of the region that triggers these problems? No worries if not, it's just a potential shortcut to reproducing the problem for ourselves.

Off the top of my head, I imagine the issue is memory leakage somewhere in Bullet. However, the stack trace itself is very likely to be showing the symptom of running out of memory rather than the root cause.
(0026313)
Gavin Hird (reporter)
2014-06-19 23:51

I am sure I can give you an oar, but this happens both on the my standalone and on the public OSGrid sims 0n two different servers. Bot instances have 28 sims running in it, but the standalone has up to significantly more rezzed content in 2 of the sims. Otherwise they are more or less identical.

Both instances are running mono 3.2.7 but I will make a couple of test running the standalone with 2.10.11 and 3.4 and report if it behaves differently.

I have uploaded the crash log for the crash on on july 18 if it can be of any use.
(0026314)
Gavin Hird (reporter)
2014-06-19 23:56

Reverted back to

Mono JIT compiler version 2.10.11 (mono-2-10/2baeee2 Wed Jan 16 16:40:16 EST 2013)
Copyright (C) 2002-2012 Novell, Inc, Xamarin, Inc and Contributors. www.mono-project.com
    TLS: normal
    SIGSEGV: normal
    Notification: kqueue
    Architecture: x86
    Disabled: none
    Misc: softdebug
    LLVM: yes(2.9svn-mono)
    GC: Included Boehm (with typed GC)

and started the standalone with bulletsim, everything else unchanged.

I could see it was allocating memory at about 1 Mb/sec and resulted in the following crash as seen in terminal, with the crash file also uploaded.

------------------

Region (root) # mono(66447,0xb1421000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mono(66447,0xb1421000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
Stacktrace:

  at (wrapper managed-to-native) OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman/BSAPICPP.PhysicsStep2 (intptr,single,int,single,int&,int&) <0xffffffff>
  at OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman.PhysicsStep (OpenSim.Region.Physics.BulletSPlugin.BulletWorld,single,int,single,int&,int&) <0x00053>
  at OpenSim.Region.Physics.BulletSPlugin.BSScene.DoPhysicsStep (single) <0x002b1>
  at OpenSim.Region.Physics.BulletSPlugin.BSScene.BulletSPluginPhysicsThread () <0x00047>
  at System.Threading.Thread.StartInternal () <0x00059>
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

    0 mono 0x000959dc mono_handle_native_sigsegv + 284
    1 mono 0x000dadcd sigabrt_signal_handler + 109
    2 libsystem_platform.dylib 0x928a1deb _sigtramp + 43
    3 ??? 0xffffffff 0x0 + 4294967295
    4 libsystem_c.dylib 0x927f029c abort + 155
    5 libc++abi.dylib 0x9251c6c9 __cxa_bad_cast + 0
    6 libc++abi.dylib 0x9253d47d _ZL25default_terminate_handlerv + 264
    7 libc++abi.dylib 0x9253ac30 _ZSt11__terminatePFvvE + 14
    8 libc++abi.dylib 0x9253a64b _ZN10__cxxabiv1L22exception_cleanup_funcE19_Unwind_Reason_CodeP17_Unwind_Exception + 0
    9 libc++.1.dylib 0x92ddeb66 _Znwm + 102
    10 libBulletSim.dylib 0x08190de9 _ZN15CProfileManager13Start_ProfileEPKc + 67
    11 libBulletSim.dylib 0x0814a489 _ZN16btCollisionWorld23computeOverlappingPairsEv + 27
    12 libBulletSim.dylib 0x0814bd04 _ZN16btCollisionWorld33performDiscreteCollisionDetectionEv + 52
    13 libBulletSim.dylib 0x081356a2 _ZN23btDiscreteDynamicsWorld28internalSingleStepSimulationEf + 136
    14 libBulletSim.dylib 0x08135ca8 _ZN23btDiscreteDynamicsWorld14stepSimulationEfif + 330
    15 libBulletSim.dylib 0x0810841d _ZN9BulletSim12PhysicsStep2EfifPiS0_ + 189
    16 libBulletSim.dylib 0x080f2879 PhysicsStep2 + 105
    17 ??? 0x080d75c0 0x0 + 135099840
    18 ??? 0x080d755c 0x0 + 135099740
    19 ??? 0x080d5b7a 0x0 + 135093114
    20 ??? 0x080d50c8 0x0 + 135090376
    21 ??? 0x04c48d72 0x0 + 79990130
    22 ??? 0x004d2861 0x0 + 5056609
    23 mono 0x0000ce02 mono_jit_runtime_invoke + 722
    24 mono 0x001a849a mono_runtime_invoke + 170
    25 mono 0x001a9d12 mono_runtime_delegate_invoke + 146
    26 mono 0x001eefb6 start_wrapper_internal + 726
    27 mono 0x001ef057 start_wrapper + 23
    28 mono 0x0023599e thread_start_routine + 206
    29 mono 0x0026fd68 GC_start_routine + 120
    30 libsystem_pthread.dylib 0x96dc25fb _pthread_body + 144
    31 libsystem_pthread.dylib 0x96dc2485 _pthread_struct_init + 0
    32 libsystem_pthread.dylib 0x96dc7cf2 thread_start + 34

Debug info from gdb:

mono(66447,0xb498b000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mono(66447,0xb529d000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
mono(66447,0xb1a2d000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
mono(66447,0xb4685000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mono(66447,0xb3a6d000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mono(66447,0xb3461000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_allocterminating with uncaught exception of type std::bad_alloc: std::bad_alloc

libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
mono(66447,0xb5eb5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mono(66447,0xb5baf000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
mono(66447,0xb61bb000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mono(66447,0xb61bb000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
mono(66447,0xb4685000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
Abort trap: 6
(0026315)
Gavin Hird (reporter)
2014-06-20 00:34

Upgraded to mono


Mono JIT compiler version 3.4.0 ((no/d4511ef Tue Mar 25 14:35:52 EDT 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
    TLS: normal
    SIGSEGV: altstack
    Notification: kqueue
    Architecture: x86
    Disabled: none
    Misc: softdebug
    LLVM: yes(3.4svn-mono-(no/e656cac)
    GC: sgen

This version creates another issue where it keeps logging messages like this: 09:09:25 - 09:25:26 - [XEngine]: Failed to save state of script door hinge.Door hinging, item UUID 2fb3580a-3278-47ff-95a0-7c45f7086057, prim UUID b8330ffb-23c4-420e-97b7-2c489beb72da in East andwest. Exception System.FormatException: The specified format '0.000000' is invalid

Also on this version it allocates memory but at a slightly lower rate of 4 Mb/sec till it crash with the following messages in terminal (crash file uploaded):


The cash file reveals that Thread# 93 crashed where these are involved in the crash:

9 libBulletSim.dylib 0x04caede9 CProfileManager::Start_Profile(char const*) + 67
10 libBulletSim.dylib 0x04c53b83 btDiscreteDynamicsWorld::stepSimulation(float, int, float) + 37
11 libBulletSim.dylib 0x04c2641d BulletSim::PhysicsStep2(float, int, float, int*, int*) + 189
12 libBulletSim.dylib 0x04c10879 PhysicsStep2 + 105


--------------------



Region (root) # mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
Stacktrace:

  at <unknown> <0xffffffff>
mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(73594,0xb16a5000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(73594,0xb5c2f000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
mono(73594,0xb5c2f000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc
mono(73594,0xb5c2f000) malloc: *** mach_vm_map(size=1048576) failed (error code=3)
*** error: can't allocate region securely
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6
(0026316)
Gavin Hird (reporter)
2014-06-20 00:47
edited on: 2014-06-20 01:50

Reverted back to mono

Mono JIT compiler version 3.2.7 ((no/40f92d5 Thu Feb 6 18:29:16 EST 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
    TLS: normal
    SIGSEGV: altstack
    Notification: kqueue
    Architecture: x86
    Disabled: none
    Misc: softdebug
    LLVM: yes(3.4svn-mono-(no/e656cac)
    GC: sgen

and started the standalone with one region only. Memory is stable over time once the sim is fully loaded, so breaking and loading two regions to see if this provokes a crash.

Loading 2 regions results in a stable memory situation.
Loading the 3rd region results in a memory allocation rate of about 200 Kb/s – I will try to load a different region and see it the result is the same. – Same result
Loading 4 regions in a 2x2 config and the memory allocation rate is now in the order of 100 Kb/s
Loading 5 regions and the memory allocation rate is back to 200 Kb/s
Loading 6 regions in a 3x2 config and the memory allocation rate is now in the order of 200 Kb/s
Loading 9 regions in a 3x3 config and the memory allocation rate is now in the order of 135 Kb/s
Loading 10 regions with the 10 adjacent to the south-west corner of the previous 3x3 config and it now allocates memory at a rate close to 400 Kb/s

Loaded all my 28 regions which are in a 4x7 configuration and the memory allocation rate is back to a steady 1 Mb/s

To me it seems like there might be something about the odd / even tiling that trips the physics engine + that tiling regions next to each other in the same simulator provokes the memory leak.

(0026420)
Gavin Hird (reporter)
2014-07-02 02:16
edited on: 2014-07-02 02:16

As an experiment I set the parameter BulletEngine = "bulletxna" in OpenSimDefaults.ini and the memory leak went away.

CPU load on idle standalone increased from in the 35+% range to 160+% range, so clearly much higher load by these libraries

(0026742)
Gavin Hird (reporter)
2014-08-13 03:28

Upgraded to Mono 3.6.0 and it still leaks memory at about 1 Mb/sec with bulletsim enabled. Otherwise everything running smoot.
(0026745)
Robert Adams (administrator)
2014-08-13 18:13

I have a hard time finding this problem because I can't reproduce it. What operating system is this on? Mac?
(0026746)
Gavin Hird (reporter)
2014-08-13 23:17

The problem has been observed on OS X 10.6.7 and for this user persisted after he upgraded to 10.9.3. All other observations by me and others have been on OS X 10.9.x.

The problem goes away if BulletEngine = "bulletxna" is set in OpenSim.ini

The problem is compounded by the number of sims in the instance and cannot be repro-ed with only one sim. For the problem to develop fast(er) I recommend having 8 sims or more in the server instance.

I can try and build a vmware virtual machine disk where the problem can be observed if you cannot repro yourself.
(0026747)
Robert Adams (administrator)
2014-08-14 07:42

I don't have a build environment for OSX. The version of BulletSim for OSX was built by another contributor. Bullet does leak memory if its internal logging is turned on (the default case). BulletXNA is a complete C# port of Bullet and is thus slower but completely within the C# managed environment.

I'll look for a host to build OSX versions of BulletSim but, for the moment, you might have to stick to ODE.
(0026748)
Gavin Hird (reporter)
2014-08-14 07:54

If it is just a matter of recompiling with other params I can possibly do it if there are reasonable instructions, or I can give you shell access to a system where you can build yourself.
(0026760)
Robert Adams (administrator)
2014-08-17 21:56

if I could have access to an OSX machine to build that would be perfect. I am just reconstructing my build environment so I can upgrade to Bullet 2.82 and, eventually, Bullet 3 (with hardware acceleration!!). Adding OSX to that build would be great. Contact me directly at misterblue@misterblue.com .
(0026761)
Luisillo_Contepomi (reporter)
2014-08-18 00:38

I have the same problem on Linux 64 Mono 2.10.8
Simulator version 8.0 (source from dist.opensimulator.org)
Grid mode 5 regions in a instance.
The problem is when I give the "shutdown" command in console, then the region crash doing a core dump and the same message reported by JeffKelley in the console.
The region never crash (by now) when is running with users in, only if I order the shutdown.
(0026763)
Robert Adams (administrator)
2014-08-18 06:26

That sounds like a slightly different problem (overflow crash on shutdown but not when running). Attaching the output of the mono crash could help. Also, if it happens immediately (you don't have to leave the regions up for a long time), turning on physics logging (look in OpenSimDefaults.ini under [BulletSim]) would provide extra log files.
(0026775)
Gavin Hird (reporter)
2014-08-20 00:21

I sent you a mail message a couple days back.

- Issue History
Date Modified Username Field Change
2014-05-18 15:07 JeffKelley New Issue
2014-05-19 15:25 justincc Note Added: 0026129
2014-05-19 16:07 JeffKelley File Added: show-scene.txt
2014-05-19 16:09 JeffKelley Note Added: 0026130
2014-05-20 11:20 Robert Adams Assigned To => Robert Adams
2014-05-20 11:20 Robert Adams Status new => assigned
2014-06-01 00:40 Gavin Hird Note Added: 0026217
2014-06-18 00:49 Gavin Hird Note Added: 0026298
2014-06-18 13:38 justincc Note Added: 0026299
2014-06-18 22:26 Gavin Hird Note Added: 0026304
2014-06-19 15:33 justincc Note Added: 0026309
2014-06-19 23:51 Gavin Hird Note Added: 0026313
2014-06-19 23:52 Gavin Hird File Added: mono-sgen_2014-06-18-095629_u2.crash
2014-06-19 23:56 Gavin Hird Note Added: 0026314
2014-06-19 23:57 Gavin Hird File Added: mono_2014-06-20-083901_u2.crash
2014-06-20 00:34 Gavin Hird Note Added: 0026315
2014-06-20 00:34 Gavin Hird File Added: mono-sgen_2014-06-20-092700_u2.crash
2014-06-20 00:47 Gavin Hird Note Added: 0026316
2014-06-20 00:54 Gavin Hird Note Edited: 0026316 View Revisions
2014-06-20 00:57 Gavin Hird Note Edited: 0026316 View Revisions
2014-06-20 01:00 Gavin Hird Note Edited: 0026316 View Revisions
2014-06-20 01:09 Gavin Hird Note Edited: 0026316 View Revisions
2014-06-20 01:14 Gavin Hird Note Edited: 0026316 View Revisions
2014-06-20 01:19 Gavin Hird Note Edited: 0026316 View Revisions
2014-06-20 01:27 Gavin Hird Note Edited: 0026316 View Revisions
2014-06-20 01:36 Gavin Hird Note Edited: 0026316 View Revisions
2014-06-20 01:49 Gavin Hird Note Edited: 0026316 View Revisions
2014-06-20 01:50 Gavin Hird Note Edited: 0026316 View Revisions
2014-07-02 02:16 Gavin Hird Note Added: 0026420
2014-07-02 02:16 Gavin Hird Note Edited: 0026420 View Revisions
2014-08-13 03:28 Gavin Hird Note Added: 0026742
2014-08-13 18:13 Robert Adams Note Added: 0026745
2014-08-13 23:17 Gavin Hird Note Added: 0026746
2014-08-14 07:42 Robert Adams Note Added: 0026747
2014-08-14 07:54 Gavin Hird Note Added: 0026748
2014-08-17 21:56 Robert Adams Note Added: 0026760
2014-08-18 00:38 Luisillo_Contepomi Note Added: 0026761
2014-08-18 06:26 Robert Adams Note Added: 0026763
2014-08-20 00:21 Gavin Hird Note Added: 0026775


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker