Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006987opensim[REGION] OpenSim Corepublic2014-01-31 08:062015-05-28 05:16
ReporterMata Hari 
Assigned ToMata Hari 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionwon't fix 
PlatformCore 2 Duo E6850OSlinuxOS VersionUbuntu 13.10
Product Versionmaster (dev code) 
Target VersionFixed in Version 
Summary0006987: Very high process memory consumption from mesh attachments (and mesh in general)
DescriptionI've noticed some odd escalating memory consumption recently and decided to monitor is closely to see if I could find out what was causing it.

I did my testing on my OSG installation (rather than my standalone): 2 regions connected to OSG, one more or less empty and the other fairly full of mesh objects (plus advanced materials on most). The default OSG ini settings are being used (flotsam cache set to memory=false and file = true possibly being the most relevant to this issue) and version is the current OSG build from 966ab21839d9cccf66b440834cd8b469f7b19e2d : [r/24256]

Upon initial sim start-up on the total process memory is approx. 400MB which seems about right compared to other sims. When I then log into one of the regions with my OSG avi this value doubles which seems somewhat high but may be normal (she is a full mesh avi wearing mesh clothing, with all including advanced mats). So for the test my "baseline" process memory usage is approximately 800MB.

I then uploaded a relatively basic mesh dress with a .dae file size of 7MB (pretty common file size for clothing item which is about 12k verts and includes its own embedded UV map, normal map, rigging -- I use custom armature slightly different than the SL one -- and weighting data). Mesh was uploaded using default LOD values that Firestorm calculates and with physics set to "minimum" (and of course the rigging and UV map) all of which contributes to final file size when the asset is stored.

I then removed all worn mesh clothing. Memory consumption didn't change (maybe this is expected behaviour?).

I then wore the mesh dress I'd just uploaded and memory consumption increased by 350MB!!! I can't imagine how an untextured 7MB mesh file suddenly became 50x greater...so now process memory is sitting at 0000006:0000001.15GB.

I then un-wore the dress and made a copy of it (I habitually do this before texturing because sometimes advanced mats can get "stuck" on a face and can't be removed...you have to re-upload the mesh). Again, no change in memory so still sitting at 1.15GB.

I then wore the copy I'd just made and process memory jumped by another 350MB to 0000006:0000001.4GB. Um...is this expected behaviour?

I then uploaded a diffuse texture as well normal and specular maps I'd made for it; edited it to apply them, and unwore the dress to "save" those changes. Again, no change in process memory (still at 1.4GB). I then re-wore it to make sure the textures had been stored properly (they had) and doing this didn't appear to have much/any impact on process memory (was ~1MB higher for the 3 texture files but I expect that happened when I initially applied them and I just didn't notice because it wasn't a major spike in consumption).

Wondering if this was a "temporary" memory usage related to new items entering the scene, I left myself standing there for a few hours without doing *anything* and process memory didn't change at all. I decided to continue this "experiment" and left myself there overnight....again no change at all in process memory usage...it held steady at 0000006:0000001.4GB for the entire 10-11 hours I left her standing there.

I then logged out. No change at all in process memory...stayed right at the 1.4GB mark and remained that way for the next 3 hours with nobody on the sim at all.

I then shut down the sim, and re-started it. Process memory went back to the expected 400MB from the previous test.

I then logged back into the same region with my avi (who was still wearing the same new dress). Process memory increased by about 340MB this time -- about 60MB less than jump when I logged in on the previous day's test with the only difference being that I was now wearing the new mesh dress instead of the mesh top and jeans I'd been wearing prior to the previous day's login.

I then removed the dress (no memory change) and then wore the top and jeans from the previous day and process memory jumped to 0000006:0000001.15GB which is the same combined total of the two outfits from the day before.

I then logged out of the sim again (and no change in memory) and then about 5 minutes later logged back in....no change in memory -- still sitting at 0000006:0000001.15GB.

Next, I "Ruthed" myself, removing all mesh entirely and wearing only the default SL avi Ruth shape, skin, etc. No change in process memory. I then logged out again (again no change in memory).

I then shut down and restarted the sim..process memory back to ~400MB as before.

I then logged in again, but of course this time as "Ruth" since that's what I'd been wearing when I logged out. Process memory barely even registered it... this time there was only maybe a 5-6MB increase in use from having "Ruth" standing there.

I then rezzed the original copy of the mesh dress on the ground and this time process memory jumped by "only" 50MB (vs the 350MB it had jumped when I attached it). It still seems a lot higher than I would have expected (7MB file > 50MB rezzed) but at least it's smaller. When I deleted that from the scene the process memory jumped another 50MB (expected for possible "undo" or "restore"?) and it remained at this higher level even after I logged out again.

One final note...I know from past experience with memory escalations for swapping mesh clothing a lot that this memory is never released....it keeps on climbing until you're using your swap and will stay like that for days. The only way to free any memory is to shut down and restart the sim.

Sorry for giving you a novel, but I wanted to be thorough in explaining what I'd done.
Steps To Reproducesee above
Additional InformationSo this makes me wonder...

1. How does a 7MB source dae become 50MB rezzed on the ground?

2. Why does the identical mesh rezzed as an attachment use 7 times more process memory than it does on the ground?

3. Why is this memory *never* released (particularly when caching is set to use flotsam file caching)?
TagsNo tags attached.
Git Revision or version number966ab21839d9cccf66b440834cd8b469f7b19e2d
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Script Engine
EnvironmentMono / Linux32
Mono Version2.10
ViewerFirestorm 4.5.1.38838 OS
Attached Files

- Relationships
related to 0007038closedorenh Unwearing mesh attachments very slow and generates [SynchronousObjectRequester] message IF mesh contains a script 

-  Notes
(0025093)
Mata Hari (reporter)
2014-01-31 08:24

Oh...and if the mesh file sizes really are so huge, it would go a long way towards explaining why teleports frequently fail for me when wearing mesh (if several hundred MB of files have to be sent before it times out). I usually have to Ruth myself, then tp, then wear the mesh again upon arrival.
(0025116)
Mata Hari (reporter)
2014-02-04 08:35

Some further data...

I've been tracking memory consumption pretty closely in the last few days since I've been uploading quite a few mesh objects. It seems that this extreme memory increase isn't consistently uniform...for no apparent reason the addition of some items to scene or rez-to-attach seems to have almost no effect on the memory and in other cases a seemingly similar mesh can eat up even more than the ratios I reported above.

Possible clues as to what *might* be contributing to this:

1. A mesh object seems to be far more likely to eat a huge amount of memory if the dae is a "linkset" of meshes rather than a single mesh where all components are joined and the more original linked objects, the more likely it seems to eat memory (I haven't done exhaustive testing on this so it's just a general observation from about 15-20 assorted uploads in the last few days).

Just to be clear about what I mean in case whoever is reading this doesn't work with mesh, if your original Blender file has 5 objects you can select them and export to dae as a single file (sort of an object linkset). Importing the file to Opensim brings in a single mesh object with each of the original 5 objects being accessible using "edit linked" which has the advantage of getting around the 8-texture limitation on a mesh without having to upload each separately and then re-assemble it in world. This is the scenario that seems more likely to produce a process memory spike. If I take that identical file and join all 5 objects to become an identical single object in Blender, then export it, it seems less likely to result in the memory spike (but then is subject to the texturing limitations...plus for some objects it can be very, very difficult to select a specific face to texture instead of using short-cut keys to select each individual member of a linkset).

2. Size doesn't seem to matter. I've uploaded a very small, relatively simple mesh object (500k dae file size) and seen it eat 30MB of process memory when attached; and I've also uploaded a 21MB dae and had it (very surprisingly!) only add 10-12MB to process memory.

3. As far as I can tell, process memory *never* goes down until the sim is shut down and restarted. When an avi leave the sim that memory is never released (even more than a day later) and the same seems to be true of when objects are deleted from a scene. I deleted the entire contents of a region in preparation to building something new on it and left it sitting empty for more than a day (when I restarted the sim). There was no reduction at all in process memory during that time, but when the sim restarted it was now almost 300MB less so presumably that's roughly the size of what I'd deleted from the scene (this was prior to any avi logins).

4. Very often (but not always) attempting to unwear a memory-eating attached mesh or deleting one that has been rezzed on the ground will result in an error in the console. Message:

08:17:29 - [LLCLIENTVIEW]: Caught exception while processing OpenMetaverse.Packets.MoveTaskInventoryPacket for Mata Hari, Document element did not appear. Line 1, position 1. at Mono.Xml2.XmlTextReader.Read () [0x00000] in <filename unknown>:0
  at System.Xml.XmlTextReader.Read () [0x00000] in <filename unknown>:0
  at System.Xml.XmlDocument.ReadNodeCore (System.Xml.XmlReader reader) [0x00000] in <filename unknown>:0
  at System.Xml.XmlDocument.ReadNode (System.Xml.XmlReader reader) [0x00000] in <filename unknown>:0
  at System.Xml.XmlDocument.Load (System.Xml.XmlReader xmlReader) [0x00000] in <filename unknown>:0
  at System.Xml.XmlDocument.LoadXml (System.String xml) [0x00000] in <filename unknown>:0
  at OpenSim.Server.Base.ServerUtils.ParseXmlResponse (System.String data) [0x00000] in <filename unknown>:0
  at OpenSim.Services.Connectors.XInventoryServicesConnector.MakeRequest (System.String method, System.Collections.Generic.Dictionary`2 sendData) [0x00000] in <filename unknown>:0
  at OpenSim.Services.Connectors.XInventoryServicesConnector.AddItem (OpenSim.Framework.InventoryItemBase item) [0x00000] in <filename unknown>:0
  at OpenSim.Region.CoreModules.ServiceConnectorsOut.Inventory.RemoteXInventoryServicesConnector.AddItem (OpenSim.Framework.InventoryItemBase item) [0x00000] in <filename unknown>:0
  at OpenSim.Region.CoreModules.ServiceConnectorsOut.Inventory.HGInventoryBroker.AddItem (OpenSim.Framework.InventoryItemBase item) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.Scene.AddInventoryItem (OpenSim.Framework.InventoryItemBase item) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.Scene.AddInventoryItem (IClientAPI remoteClient, OpenSim.Framework.InventoryItemBase item) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.Scene.MoveTaskInventoryItem (IClientAPI remoteClient, UUID folderId, OpenSim.Region.Framework.Scenes.SceneObjectPart part, UUID itemId) [0x00000] in <filename unknown>:0
  at OpenSim.Region.Framework.Scenes.Scene.ClientMoveTaskInventoryItem (IClientAPI remoteClient, UUID folderId, UInt32 primLocalId, UUID itemId) [0x00000] in <filename unknown>:0
  at OpenSim.Region.ClientStack.LindenUDP.LLClientView.HandleMoveTaskInventory (IClientAPI sender, OpenMetaverse.Packets.Packet Pack) [0x00000] in <filename unknown>:0
  at OpenSim.Region.ClientStack.LindenUDP.LLClientView.ProcessSpecificPacketAsync (System.Object state) [0x00000] in <filename unknown>:0
(0025392)
Mata Hari (reporter)
2014-03-07 10:23

This issue continues as of r/24408. After uploading a single ~10MB mesh dress I had a series of different textures I wanted to apply to it. When I reached the 6th one the sim began giving "out of memory" exceptions and would not save any changes until I shut down and restarted it. Prior to beginning the work I had about 800MB process memory used. I didn't track it on a per-item level but after uploading the mesh and attempting to texture the 6th, the total process memory was over 8GB!

I use flotsam file cache only (no flotsam memory cache) and was wearing the dress each time I textured it.
(0028473)
Mata Hari (reporter)
2015-05-28 05:16

Closing as unresolved - won't fix

- Issue History
Date Modified Username Field Change
2014-01-31 08:06 Mata Hari New Issue
2014-01-31 08:24 Mata Hari Note Added: 0025093
2014-02-04 07:11 aiaustin Summary Veyr high process memory consumption from mesh attachments (and mesh in general) => Very high process memory consumption from mesh attachments (and mesh in general)
2014-02-04 08:35 Mata Hari Note Added: 0025116
2014-03-07 10:23 Mata Hari Note Added: 0025392
2014-03-27 09:23 Mata Hari Relationship added related to 0007038
2015-05-28 05:14 Mata Hari Status new => resolved
2015-05-28 05:14 Mata Hari Resolution open => won't fix
2015-05-28 05:14 Mata Hari Assigned To => Mata Hari
2015-05-28 05:16 Mata Hari Note Added: 0028473
2015-05-28 05:16 Mata Hari Status resolved => closed


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker