Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006766opensim[REGION] Physics Enginespublic2013-09-11 23:452014-07-29 13:42
Reporterkenvc 
Assigned ToRobert Adams 
PriorityimmediateSeveritycrashReproducibilityalways
StatusclosedResolutionfixed 
PlatformDual Quad Core Xeon 32GBOSWindowsOS VersionServer 2012 64
Product Versionmaster (dev code) 
Target VersionFixed in Versionmaster (dev code) 
Summary0006766: Bulletsim: Windows & Ubuntu - Using massive amount more memory compared to ODE sometimes causing Out of Memory crashes
DescriptionThe latest round of changes in last 3-4 days related to Bulletsim appears to be causing opensim to use significantly more memory. I have been using the 32 bit launcher patched to use 4gb memory and all has been good for a very long time until this week. Many of the instances I have been running are now crashing with out of memory errors.

See the log info in "Additional Information" below. The last out of memory error message repeats over and over and windows pops up a messagebox that says opensim has caused an exception and needs to be closed.
Steps To ReproduceUse the latest dev master with bulletsim and the 32 bit launcher
Additional InformationInitialized 0 script instances in Polynesia_70
2013-09-11 16:05:20,924 INFO - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Starting inbound packet processing for the LLUDP server in asynchronous mode with UsePools = True
2013-09-11 16:05:20,940 INFO - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Starting outbound packet processing for the LLUDP server
2013-09-11 16:05:20,955 ERROR - OpenSim.Region.Physics.BulletSPlugin.BSScene [BULLETS SCENE]: ProcessPostTaintTaints: TerrainManager.AddMegaRegionChild<1536, 1024, 0>-2: Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman.BSAPICPP.CreateMeshShape2(IntPtr world, Int32 indicesCount, Int32[] indices, Int32 verticesCount, Single[] vertices)
   at OpenSim.Region.Physics.BulletSPlugin.BSAPIUnman.CreateMeshShape(BulletWorld world, Int32 indicesCount, Int32[] indices, Int32 verticesCount, Single[] vertices)
   at OpenSim.Region.Physics.BulletSPlugin.BSTerrainMesh..ctor(BSScene physicsScene, Vector3 regionBase, UInt32 id, Single[] initialMap, Vector3 minCoords, Vector3 maxCoords)
   at OpenSim.Region.Physics.BulletSPlugin.BSTerrainManager.BuildPhysicalTerrain(Vector3 terrainRegionBase, UInt32 id, Single[] heightMap, Vector3 minCoords, Vector3 maxCoords)
   at OpenSim.Region.Physics.BulletSPlugin.BSTerrainManager.UpdateTerrain(UInt32 id, Single[] heightMap, Vector3 minCoords, Vector3 maxCoords)
   at OpenSim.Region.Physics.BulletSPlugin.BSTerrainManager.<>c__DisplayClass4.<AddMegaRegionChildTerrain>b__3()
   at OpenSim.Region.Physics.BulletSPlugin.BSScene.ProcessPostTaintTaints()
2013-09-11 16:05:21,049 ERROR - OpenSim.Application [APPLICATION]:
APPLICATION EXCEPTION DETECTED: System.UnhandledExceptionEventArgs

Exception: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
   at System.Threading.Thread.Start()
   at OpenSim.Framework.Monitoring.Watchdog.StartThread(ThreadStart start, String name, ThreadPriority priority, Boolean isBackground, Boolean alarmIfTimeout, Func`1 alarmMethod, Int32 timeout)
   at OpenSim.Framework.Monitoring.Watchdog.StartThread(ThreadStart start, String name, ThreadPriority priority, Boolean isBackground, Boolean alarmIfTimeout)
   at OpenSim.Region.Framework.Scenes.Scene.Heartbeat()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

Application is terminating: True
TagsNo tags attached.
Git Revision or version numberr/23948
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Environment.NET / Windows64
Mono VersionNone
ViewerN/A
Attached Files

- Relationships

-  Notes
(0024331)
hack13 (reporter)
2013-09-12 00:43

I am unable to replicate this in mono 2.10.8
(0024335)
Robert Adams (administrator)
2013-09-12 08:53

Sadly, I do see memory creeping up with the latest BulletSim and 2.10.8.1:
radams1@m1 ~/logs> grep "Process memory" OpenSim.log
Process memory : 477 MB
Process memory : 525 MB
Process memory : 568 MB
Process memory : 594 MB
Process memory : 626 MB
Process memory : 655 MB
Process memory : 688 MB
Process memory : 734 MB
Process memory : 767 MB
Process memory : 794 MB
Process memory : 820 MB
Process memory : 863 MB
Process memory : 904 MB
Process memory : 947 MB
Process memory : 988 MB
Process memory : 1023 MB
Process memory : 1051 MB
Process memory : 1095 MB
radams1@m1 ~/logs> mono --version
Mono JIT compiler version 2.10.8.1 (Debian 2.10.8.1-1ubuntu2.2)
...
radams1@m1 ~/logs>
(0024337)
kenvc (reporter)
2013-09-12 08:58

Yes, I have seen it climb and and have an out of memory exception on sims where there is no activity and no one has even logged in or TP'd to that sim and the sim does not have all that many prims.
(0024338)
Robert Adams (administrator)
2013-09-12 13:12

I found that Bullet's performance profiler was accidentally left enabled. It has a known bug of leaking memory.

I just checked in new DLLs and SOs with the profiler disabled. The leaking seems to be fixed in my testing. Check it out on your systems.
(0024339)
kenvc (reporter)
2013-09-12 15:19

Robert, the code change helped for sure. Now I am getting crashes only on the two 25 sim megas when running with the 32 bit launcher. The really odd thing is that the 32 bit launcher is now using just as much memory as the 64 bit opensim. In the past the 32 bit launcher consumed about 25-50% less memory than the 64 bit exe. This part might not be related to bulletsim. I will switch back to ODE and run the same memory tests again and see what we have.
(0024344)
kenvc (reporter)
2013-09-12 15:57
edited on: 2013-09-23 11:55

With the latest Bulletsim code, Bulletsim is using 27% more memory than ODE when used with 64 bit opensim. The real shocker is that Bulletsim is using 290% more memory than ODE when used with 32 bit opensim!

Because of all the out of memory crashes, I reluctantly had to switch back to ODE after having run Bulletsim for the last few months with no real issues. I'll be happy to switch back to Bulletsim once this issue is cleared.

(0024407)
kenvc (reporter)
2013-09-26 12:44
edited on: 2013-09-26 13:28

This issue may be related to mantis 6599. Testing the fix in 6599 now...
After adding TerrainMeshMagnification = 1 in [BulletSim] section, the memory issue is significantly better, but still overall about 25% higher than ODE uses. Will continue to run BulletSim with this setting on my test grid for a while before trying it again on my sims on OSG to make sure the memory usage does not grow with time and that no crashes happen like it did with the default setting.

(0024489)
kenvc (reporter)
2013-10-16 17:43

I am happy to see Bulletsim as the default Physics engine, but the massive increase in memory consumption when Bulletsim is enabled has still not been resolved.

When ODE is enabled, about 60% of the 32GB server memory is consumed by Robust and all my sims. When Bulletsim is enabled on all sims, the memory consumption goes up to 97% of the 32 gig during startup, then settles down to 94% after everything is started. If a few AVs come into the sims, its going to hit 100% on memory usage. Is there any way to change a setting to make Bulletsim use even less memory than the suggestion that I already did from mantis 6599?
(0024490)
kenvc (reporter)
2013-10-16 18:48

This issue appears to happen when running Bulletsim with Windows. Nebadon says he has not noticed any memory increase when using Bulletsim under Linux, but under Windows it is using about 40% more memory at least compared to ODE.
(0024492)
Gwyneth Llewelyn (reporter)
2013-10-16 19:13

I do see an almost 100% increase of memory under Ubuntu 13.04 and Mono 2.10.8.1 — this was tested on a two-region simulator with just a handful of prims.
(0024497)
kenvc (reporter)
2013-10-17 09:41

Gwyneth,

It is very odd that Nebadon is not seeing any increase with Linux, but he is not using Ubuntu. At any rate, a 40-100% memory increase when using Bulletsim is an indication that something is very wrong.
(0024500)
Gwyneth Llewelyn (reporter)
2013-10-17 12:26

Ken,

I've been documenting my struggles with BulletSim on a separate thread (http://opensimulator.org/mantis/view.php?id=6789 [^]) — more related to CPU consumption, which worries me far more, than with memory issues (because I have plenty to spare). As far as I can see it, the issue is strictly limited to that particular server and my specific configuration, since I've tried to replicate the issue without any luck, even after several Ubuntu upgrades — from 12.04 to 12.10 to 13.04. I'm using Mono 2.10.8.1 because upgrading it to a more recent version is not straightforward; so maybe Nebadon is using a different distribution which has a more recent version of Mono? This could be merely a Mono issue.

Whatever the reason, under *some* circumstances, apparently, BulletSim consumes a massive amount of memory/CPU without any obvious errors, and, so far, I have been very unlucky to track it down — unless it's the server's hardware which is faulty (but then again, why doesn't it have any issue with ODE whatsoever? In fact, since 0.7.6 and progressing to 0.8.0, CPU & memory consumption seem to be getting lower and lower with each release).

Robert suggested elsewhere (http://opensimulator.org/mantis/view.php?id=6806 [^] — regarding running BulletSim under Mac OS X) that there might be some issues with the dynamic libraries in C++, but that ought only to affect Mac users — not Linux.
(0024510)
kenvc (reporter)
2013-10-17 12:57

Gwyneth,

I am using the unmanaged version of Bulletsim which I think is the default. Which flavor of Bulletsim are you using?
(0024511)
kenvc (reporter)
2013-10-17 13:15

In case it gives any additional clues, I have two 25 sim megas, two 4 sim megas, and the other 60 or so sims are single sims running about 5 sims per instance.
(0024516)
Gwyneth Llewelyn (reporter)
2013-10-17 18:27
edited on: 2013-10-17 18:29

I'm using the unmanaged version as well.

In my case, I have 6 instances, running from 2 to 6 single regions each (they're spread between instances according to the number of prims each; roughly speaking, each instance has less than 100,000 prims).

But for establishing memory consumption, I generally just launch a single instance — either with 2 regions with almost no prims, or 2 regions with a total of around 6,000 prims. To be honest, I haven't evaluated the memory consumption on *all* instances individually and taken a correlation with the number of regions, prims, avatars, etc. Globally speaking, and looking from the "outside" (i.e. free, top, and similar tools) it's quite visible that the overall memory consumption is almost twice with BulletSim compared to ODE, and this is consistent with what OpenSim reports on the console.

But I should state again that this doesn't worry me too much — I have enough memory for all that! — it's really the CPU consumption that brings an otherwise "empty" server to its knees — with just a single instance, much less when running with all six. Interestingly enough, the viewer does not report a reduction on Physical FPS or Sim FPS, but you can visually see how it struggles to even maintain a single avatar walking around the place.

(0024524)
kenvc (reporter)
2013-10-18 11:18

It is looking like at least a temporary workaround for this issue is to use the following setting:

[BulletSim]
TerrainImplementation=0

My memory usage is now the same as ODE and when using Neb's racer, it actually seems a bit more responsive and I have not found any drawbacks to using this setting at this time.
(0024542)
kenvc (reporter)
2013-10-19 13:17

Just tested r/2396 with the latest Bulletsim changes to help resolve this issue. I set TerrainImplementation = 1 and restarted all sims. Memory usage is within 4% or what it was with ODE and what it was with TerrainImplementation = 0. If this is not the final fix, it is close to it. Will wait until more testing has been done before marking this as resolved.
(0024545)
Gwyneth Llewelyn (reporter)
2013-10-20 06:15

Crossposted from: http://opensimulator.org/mantis/view.php?id=6789 [^]

One instance (out of six) in HG mode, with two regions and a total of 13000+ prims, latest build from git (with the new unmanaged library), running under Ubuntu 13.10, Mono 2.10.8.1:

Reference: With ODE:

Heap allocated to OpenSim : 145 MB
Last heap allocation rate : 0.375 MB/s
Average heap allocation rate: 0.068 MB/s
Process memory : 261 MB

With BulletSim, TerrainImplementation=0 and TerrainMeshMagnification = 1:

Heap allocated to OpenSim : 198 MB
Last heap allocation rate : 0.064 MB/s
Average heap allocation rate: 0.101 MB/s
Process memory : 345 MB

With BulletSim, TerrainMeshMagnification = 1:

Heap allocated to OpenSim : 199 MB
Last heap allocation rate : 0.069 MB/s
Average heap allocation rate: 0.133 MB/s
Process memory : 390 MB

So memory consumption is now 25% over ODE instead of 33%, which is certainly an improvement over 40% or 100%! (I think that memory consumption under ODE also dropped a bit with the upgrades of both Ubuntu and OpenSimulator)
(0024548)
nebadon (administrator)
2013-10-20 08:43

I tried TerrainImplementation=0 on my 64 Bit Linux simulator where I am seeing crazy high CPU while driving my racer and saw no change to cpu or memory what so ever. using Mono 3.2.3
(0024549)
Gwyneth Llewelyn (reporter)
2013-10-20 11:19

@Nebadon, when you're *not* driving your racer, what is the difference in CPU consumption compared to ODE?

My tests are actually made without avatars on the scene.
(0024581)
Robert Adams (administrator)
2013-10-28 07:36

Is the BulletSim memory problem resolved? (The CPU thing is in another Mantis)
(0024582)
kenvc (reporter)
2013-10-28 07:41

Robert,
Bulletsim is still using more memory than ODE by approximately 25% but it was much worse than that until your last round of changes. When your system already has the max memory you can install in it, this extra 25% over ODE can put it over the limit and then the only choice to shut down some sims or use a 2nd computer... which is the situation I am in.

So in summary, the issue is much improved, but anything you could do to get the memory use down to what ODE uses on regular and megas would be very helpful.
(0024590)
Gwyneth Llewelyn (reporter)
2013-10-30 16:38

I can confirm the 25% more memory usage, it's about that in my grid as well.
(0025056)
kenvc (reporter)
2014-01-25 10:33

Bulletsim is still using more memory than ODE, especially when used with the 32 bit launcher, but it is not that much more (about 25%) after Roberts last changes. I am going to consider this resolved at least the best it can be done.

- Issue History
Date Modified Username Field Change
2013-09-11 23:45 kenvc New Issue
2013-09-12 00:13 kenvc Description Updated View Revisions
2013-09-12 00:43 hack13 Note Added: 0024331
2013-09-12 08:51 Robert Adams Assigned To => Robert Adams
2013-09-12 08:51 Robert Adams Status new => assigned
2013-09-12 08:53 Robert Adams Note Added: 0024335
2013-09-12 08:58 kenvc Note Added: 0024337
2013-09-12 13:12 Robert Adams Note Added: 0024338
2013-09-12 15:19 kenvc Note Added: 0024339
2013-09-12 15:57 kenvc Note Added: 0024344
2013-09-12 16:09 kenvc Note Edited: 0024344 View Revisions
2013-09-23 11:55 kenvc Note Edited: 0024344 View Revisions
2013-09-26 12:44 kenvc Relationship added related to 0006599
2013-09-26 12:44 kenvc Note Added: 0024407
2013-09-26 13:28 kenvc Note Edited: 0024407 View Revisions
2013-10-16 17:43 kenvc Note Added: 0024489
2013-10-16 17:50 kenvc Git Revision or version number r/23647 => r/23948
2013-10-16 17:50 kenvc Priority urgent => immediate
2013-10-16 17:50 kenvc Summary Bulletsim - Out of memory exception errors on many instances with last round of Bulletsim updates => Bulletsim - Using massive amount more memory compared to ODE sometimes causing Out of Memory crashes
2013-10-16 18:48 kenvc Note Added: 0024490
2013-10-16 18:49 kenvc Summary Bulletsim - Using massive amount more memory compared to ODE sometimes causing Out of Memory crashes => Bulletsim with Windows - Using massive amount more memory compared to ODE sometimes causing Out of Memory crashes
2013-10-16 19:13 Gwyneth Llewelyn Note Added: 0024492
2013-10-17 09:41 kenvc Note Added: 0024497
2013-10-17 12:26 Gwyneth Llewelyn Note Added: 0024500
2013-10-17 12:57 kenvc Note Added: 0024510
2013-10-17 12:58 kenvc Summary Bulletsim with Windows - Using massive amount more memory compared to ODE sometimes causing Out of Memory crashes => Bulletsim: Windows & Ubuntu - Using massive amount more memory compared to ODE sometimes causing Out of Memory crashes
2013-10-17 13:15 kenvc Note Added: 0024511
2013-10-17 18:27 Gwyneth Llewelyn Note Added: 0024516
2013-10-17 18:29 Gwyneth Llewelyn Note Edited: 0024516 View Revisions
2013-10-18 11:18 kenvc Note Added: 0024524
2013-10-18 21:14 kenvc Relationship deleted related to 0006599
2013-10-19 13:17 kenvc Note Added: 0024542
2013-10-20 06:15 Gwyneth Llewelyn Note Added: 0024545
2013-10-20 08:43 nebadon Note Added: 0024548
2013-10-20 11:19 Gwyneth Llewelyn Note Added: 0024549
2013-10-28 07:36 Robert Adams Note Added: 0024581
2013-10-28 07:41 kenvc Note Added: 0024582
2013-10-30 16:38 Gwyneth Llewelyn Note Added: 0024590
2014-01-25 10:33 kenvc Note Added: 0025056
2014-01-25 10:33 kenvc Status assigned => resolved
2014-01-25 10:33 kenvc Fixed in Version => master (dev code)
2014-01-25 10:33 kenvc Resolution open => fixed
2014-07-29 13:42 chi11ken Status resolved => closed


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker