MantisBT - opensim
View Issue Details
0005640opensim[REGION] OpenSim Corepublic2011-08-14 02:082019-02-05 12:27
mewtwo0641 
mewtwo0641 
normalmajoralways
closedfixed 
master (dev code) 
master (dev code) 
r/16513
Standalone (Multiple Regions)
ODE
.NET / Windows32, .NET / Windows64
None
All
0005640: Out of Memory errors when saving/loading IAR/OAR, logging in / logging out
I am recently running into a rash of "Out Of Memory" exceptions when ever I try to delete/take into inventory multiple objects, save IARs/OARs, or when FlotsamCache is being populated on disk.

Not sure why this would be as I have 1.5 gigs of free ram available to my system after OpenSim is all loaded up and my page file usage is 1.78 Gigs out of 4.0 gigs.

Also tried the latest commit (r/16513) with the fix in it for a few hours I don't notice out of memory errors anymore at least when deleting a massive amount of prims from in world; but I still seem to be running into them almost immediately when ever I try to save / load an IAR (even IARs of only a few small objects.) and OARs as well. I also noticed out of memory errors when an avatar tries to log in or out.

Not sure if related to mantis 5634 or not.

Error on logout:

2011-08-13 13:26:16,515 ERROR - OpenSim.Region.ClientStack.LindenUDP.LLClientView [LLCLIENTVIEW]: Caught exception while processing OpenMetaverse.Packets.LogoutRequestPacket for mewtwo064 Alito, Exception of type 'System.OutOfMemoryException' was thrown. at System.String.Concat(String str0, String str1)
   at OpenMetaverse.Utils.StringToBytes(String str)
   at OpenSim.Region.CoreModules.Avatar.Attachments.AttachmentsModule.UpdateKnownItem(IClientAPI remoteClient, SceneObjectGroup grp, UUID itemID, UUID agentID) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Attachments\AttachmentsModule.cs:line 532
   at OpenSim.Region.Framework.Scenes.ScenePresence.SaveChangedAttachments() in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\Framework\Scenes\ScenePresence.cs:line 3889
   at OpenSim.Region.Framework.Scenes.Scene.RemoveClient(UUID agentID, Boolean closeChildAgents) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\Framework\Scenes\Scene.cs:line 3149
   at OpenSim.Region.ClientStack.LindenUDP.LLClientView.Close() in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLClientView.cs:line 515
   at OpenSim.Region.ClientStack.LindenUDP.LLUDPServer.RemoveClient(LLUDPClient udpClient) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLUDPServer.cs:line 992
   at OpenSim.Region.ClientStack.LindenUDP.LLUDPServer.LogoutHandler(IClientAPI client) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLUDPServer.cs:line 1300
   at System.Action`1.Invoke(T obj)
   at OpenSim.Region.ClientStack.LindenUDP.LLClientView.Logout(IClientAPI client) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLClientView.cs:line 11181
   at OpenSim.Region.ClientStack.LindenUDP.LLClientView.HandleLogout(IClientAPI client, Packet packet) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLClientView.cs:line 11165
   at OpenSim.Region.ClientStack.LindenUDP.LLClientView.ProcessSpecificPacketAsync(Object state) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLClientView.cs:line 678
2011-08-13 13:27:21,718 WARN - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Ack timeout, disconnecting d600c5f6-fbba-4146-8ab4-0776538b9705

Error saving IAR:

2011-08-15 05:05:32,406 ERROR - OpenSim.Region.Framework.Scenes.UuidGatherer [UUID GATHERER]: Failed to gather uuids for asset id 6d7e0d3b-860c-4976-8175-2588af71b43c, type Object
2011-08-15 05:05:32,406 ERROR - OpenSim.Application Command error: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.String.CreateStringFromEncoding(Byte* bytes, Int32 byteLength, Encoding encoding)
   at System.Text.UTF8Encoding.GetString(Byte[] bytes, Int32 index, Int32 count)
   at OpenMetaverse.Utils.BytesToString(Byte[] bytes)
   at OpenSim.Region.Framework.Scenes.UuidGatherer.GetSceneObjectAssetUuids(UUID sceneObjectUuid, IDictionary`2 assetUuids) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\Framework\Scenes\UuidGatherer.cs:line 300
   at OpenSim.Region.Framework.Scenes.UuidGatherer.GatherAssetUuids(UUID assetUuid, AssetType assetType, IDictionary`2 assetUuids) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\Framework\Scenes\UuidGatherer.cs:line 119
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiveWriteRequest.SaveInvItem(InventoryItemBase inventoryItem, String path, Dictionary`2 options, IUserAccountService userAccountService) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiveWriteRequest.cs:line 166
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiveWriteRequest.SaveInvFolder(InventoryFolderBase inventoryFolder, String path, Boolean saveThisFolderItself, Dictionary`2 options, IUserAccountService userAccountService) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiveWriteRequest.cs:line 202
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiveWriteRequest.SaveInvFolder(InventoryFolderBase inventoryFolder, String path, Boolean saveThisFolderItself, Dictionary`2 options, IUserAccountService userAccountService) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiveWriteRequest.cs:line 197
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiveWriteRequest.Execute(Dictionary`2 options, IUserAccountService userAccountService) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiveWriteRequest.cs:line 322
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiverModule.ArchiveInventory(Guid id, String firstName, String lastName, String invPath, String pass, String savePath, Dictionary`2 options) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiverModule.cs:line 220
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiverModule.HandleSaveInvConsoleCommand(String module, String[] cmdparams) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiverModule.cs:line 432
   at OpenSim.Framework.Console.Commands.Resolve(String[] cmd) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Framework\Console\CommandConsole.cs:line 373
   at OpenSim.Framework.Console.LocalConsole.ReadLine(String p, Boolean isCommand, Boolean e) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Framework\Console\LocalConsole.cs:line 480
   at OpenSim.Framework.Console.CommandConsole.Prompt() in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Framework\Console\CommandConsole.cs:line 592
   at OpenSim.Application.Main(String[] args) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\Application\Application.cs:line 279
No tags attached.
txt config.txt (13,707) 2011-08-17 01:26
http://opensimulator.org/mantis/file_download.php?file_id=2753&type=bug
Issue History
2011-08-14 02:08mewtwo0641New Issue
2011-08-14 02:08mewtwo0641Git Revision => r/16513
2011-08-14 02:08mewtwo0641Run Mode => Standalone (Multiple Regions)
2011-08-14 02:08mewtwo0641Physics Engine => ODE
2011-08-14 02:08mewtwo0641Environment => .NET / Windows32, .NET / Windows64
2011-08-14 02:08mewtwo0641Mono Version => None
2011-08-14 02:08mewtwo0641Viewer => All
2011-08-14 14:35mewtwo0641Description Updated
2011-08-15 03:09mewtwo0641Description Updated
2011-08-15 03:11mewtwo0641Description Updated
2011-08-15 03:12mewtwo0641Description Updated
2011-08-16 13:21justinccNote Added: 0019611
2011-08-16 13:21justinccStatusnew => feedback
2011-08-17 01:26mewtwo0641Note Added: 0019615
2011-08-17 01:26mewtwo0641File Added: config.txt
2011-08-29 01:01aiaustinNote Added: 0019778
2011-08-29 01:03aiaustinNote Edited: 0019778
2011-08-29 01:09aiaustinNote Edited: 0019778
2011-08-29 01:09aiaustinNote Edited: 0019778
2019-01-19 04:36mewtwo0641Note Added: 0033846
2019-01-19 04:36mewtwo0641Statusfeedback => new
2019-01-19 04:36mewtwo0641Statusnew => resolved
2019-01-19 04:36mewtwo0641Fixed in Version => master (dev code)
2019-01-19 04:36mewtwo0641Resolutionopen => fixed
2019-01-19 04:36mewtwo0641Assigned To => mewtwo0641
2019-02-05 12:27BillBlightNote Added: 0034313
2019-02-05 12:27BillBlightStatusresolved => closed

Notes
(0019611)
justincc   
2011-08-16 13:21   
Are you sure that MemoryCacheEnabled = false in your FlotsamCache.ini? If so, please paste the results of "config show" or "config save".

Also, how much memory is on the machine that you're using, and what memory usage is show when you execute "show stats" at different points (e.g. before loading an iar/oar, afterwards, etc.)
(0019615)
mewtwo0641   
2011-08-17 01:26   
In my FlotsamCache.ini I have MemoryCacheEnabled = false and FileCacheEnabled = true. On one system Running 3 GB of RAM with 8 GB page file set up on my system and on my other system 4 GB of RAM with 8 GB page file.

Test before an Avatar logs in
------------------------------
show stats command:
Memory usage before IAR save is 225 MB
Memory usage after IAR save is 277 MB

task manager:
Opensim.exe Mem Usage: Appx 996 MB, Virtual Memory Size: Appx 1 GB


Test after an Avatar logs in:
-----------------------------

show stats command:
Memory usage before IAR save is 254 MB
Memory usage after IAR save is 246 MB

task manager:
Opensim.exe Mem Usage: Appx 523 MB, Virtual Memory Size: Appx 1 GB

See attached file for config save results please.
(0019778)
aiaustin   
2011-08-29 01:01   
(edited on: 2011-08-29 01:09)
I am seeing this too on one of the Openvue region servers that is running r/16521 that has been running for some time.. perhaps over a week. I am backing up oar files and saving the terrain for each region.

Just restarting the OpenSim.exe allowed the regions to all be saved as OARs successfuly without the out of memory errors.

Note also typo on error line text.. "since asset requster" - should be "requester"

I will update to latest 0.7.2 dev master, which the core Openvue ROBUST and central region servers is on, as I believe a memory leak was fixed since r/16521.

2011-08-29 08:57:16,466 ERROR - OpenSim.Region.CoreModules.World.Archiver.AssetsRequest [ARCHIVER]: Terminating archive creation since asset requster callback failed with System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

   at System.IO.MemoryStream.set_Capacity(Int32 value)

   at System.IO.MemoryStream.EnsureCapacity(Int32 value)

   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)

   at System.IO.BinaryWriter.Write(Single value)

   at OpenSim.Region.CoreModules.World.Terrain.FileLoaders.RAW32.SaveStream(Stream s, ITerrainChannel map) in d:\Temp\OpenSim\Region\CoreModules\World\Terrain\FileLoaders\RAW32.cs:line 156

   at OpenSim.Region.CoreModules.World.Terrain.TerrainModule.SaveToStream(String filename, Stream stream) in d:\Temp\OpenSim\Region\CoreModules\World\Terrain\TerrainModule.cs:line 364

   at OpenSim.Region.CoreModules.World.Archiver.ArchiveWriteRequestExecution.Save(ICollection`1 assetsFoundUuids, ICollection`1 assetsNotFoundUuids) in d:\Temp\OpenSim\Region\CoreModules\World\Archiver\ArchiveWriteRequestExecution.cs:line 134

   at OpenSim.Region.CoreModules.World.Archiver.ArchiveWriteRequestExecution.ReceivedAllAssets(ICollection`1 assetsFoundUuids, ICollection`1 assetsNotFoundUuids) in d:\Temp\OpenSim\Region\CoreModules\World\Archiver\ArchiveWriteRequestExecution.cs:line 88

   at OpenSim.Region.CoreModules.World.Archiver.AssetsRequest.PerformAssetsRequestCallback(Object o) in d:\Temp\OpenSim\Region\CoreModules\World\Archiver\AssetsRequest.cs:line 302

2011-08-29 08:57:16,535 DEBUG - OpenSim.Region.CoreModules.World.Archiver.AssetsRequest [ARCHIVER]: Successfully added 10 assets (0 assets notified missing)

2011-08-29 08:57:16,538 INFO - OpenSim.Region.CoreModules.World.Archiver.ArchiveWriteRequestExecution [ARCHIVER]: Added region settings to archive.

2011-08-29 08:57:16,541 INFO - OpenSim.Region.CoreModules.World.Archiver.ArchiveWriteRequestExecution [ARCHIVER]: Added parcel settings to archive.

2011-08-29 08:57:16,561 INFO - OpenSim.Region.CoreModules.World.Archiver.ArchiveWriteRequestExecution [ARCHIVER]: Added terrain information to archive.

2011-08-29 08:57:16,807 ERROR - OpenSim.Region.CoreModules.World.Archiver.AssetsRequest [ARCHIVER]: Terminating archive creation since asset requster callback failed with System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

   at System.IO.MemoryStream.set_Capacity(Int32 value)

   at System.IO.MemoryStream.EnsureCapacity(Int32 value)

   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)

   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)

   at System.IO.StreamWriter.Write(String value)

   at System.Xml.XmlTextEncoder.Write(String text)

(0033846)
mewtwo0641   
2019-01-19 04:36   
Haven't seen this issue for a very long time; fixed by master
(0034313)
BillBlight   
2019-02-05 12:27   
Old Issues, closed, can be reopened if they still exist