Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0005640opensim[REGION] OpenSim Corepublic2011-08-14 02:082019-02-05 12:27
Reportermewtwo0641 
Assigned Tomewtwo0641 
PrioritynormalSeveritymajorReproducibilityalways
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Versionmaster (dev code) 
Target VersionFixed in Versionmaster (dev code) 
Summary0005640: Out of Memory errors when saving/loading IAR/OAR, logging in / logging out
DescriptionI am recently running into a rash of "Out Of Memory" exceptions when ever I try to delete/take into inventory multiple objects, save IARs/OARs, or when FlotsamCache is being populated on disk.

Not sure why this would be as I have 1.5 gigs of free ram available to my system after OpenSim is all loaded up and my page file usage is 1.78 Gigs out of 4.0 gigs.

Also tried the latest commit (r/16513) with the fix in it for a few hours I don't notice out of memory errors anymore at least when deleting a massive amount of prims from in world; but I still seem to be running into them almost immediately when ever I try to save / load an IAR (even IARs of only a few small objects.) and OARs as well. I also noticed out of memory errors when an avatar tries to log in or out.

Not sure if related to mantis 5634 or not.

Error on logout:

2011-08-13 13:26:16,515 ERROR - OpenSim.Region.ClientStack.LindenUDP.LLClientView [LLCLIENTVIEW]: Caught exception while processing OpenMetaverse.Packets.LogoutRequestPacket for mewtwo064 Alito, Exception of type 'System.OutOfMemoryException' was thrown. at System.String.Concat(String str0, String str1)
   at OpenMetaverse.Utils.StringToBytes(String str)
   at OpenSim.Region.CoreModules.Avatar.Attachments.AttachmentsModule.UpdateKnownItem(IClientAPI remoteClient, SceneObjectGroup grp, UUID itemID, UUID agentID) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Attachments\AttachmentsModule.cs:line 532
   at OpenSim.Region.Framework.Scenes.ScenePresence.SaveChangedAttachments() in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\Framework\Scenes\ScenePresence.cs:line 3889
   at OpenSim.Region.Framework.Scenes.Scene.RemoveClient(UUID agentID, Boolean closeChildAgents) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\Framework\Scenes\Scene.cs:line 3149
   at OpenSim.Region.ClientStack.LindenUDP.LLClientView.Close() in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLClientView.cs:line 515
   at OpenSim.Region.ClientStack.LindenUDP.LLUDPServer.RemoveClient(LLUDPClient udpClient) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLUDPServer.cs:line 992
   at OpenSim.Region.ClientStack.LindenUDP.LLUDPServer.LogoutHandler(IClientAPI client) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLUDPServer.cs:line 1300
   at System.Action`1.Invoke(T obj)
   at OpenSim.Region.ClientStack.LindenUDP.LLClientView.Logout(IClientAPI client) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLClientView.cs:line 11181
   at OpenSim.Region.ClientStack.LindenUDP.LLClientView.HandleLogout(IClientAPI client, Packet packet) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLClientView.cs:line 11165
   at OpenSim.Region.ClientStack.LindenUDP.LLClientView.ProcessSpecificPacketAsync(Object state) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\ClientStack\Linden\UDP\LLClientView.cs:line 678
2011-08-13 13:27:21,718 WARN - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Ack timeout, disconnecting d600c5f6-fbba-4146-8ab4-0776538b9705

Error saving IAR:

2011-08-15 05:05:32,406 ERROR - OpenSim.Region.Framework.Scenes.UuidGatherer [UUID GATHERER]: Failed to gather uuids for asset id 6d7e0d3b-860c-4976-8175-2588af71b43c, type Object
2011-08-15 05:05:32,406 ERROR - OpenSim.Application Command error: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.String.CreateStringFromEncoding(Byte* bytes, Int32 byteLength, Encoding encoding)
   at System.Text.UTF8Encoding.GetString(Byte[] bytes, Int32 index, Int32 count)
   at OpenMetaverse.Utils.BytesToString(Byte[] bytes)
   at OpenSim.Region.Framework.Scenes.UuidGatherer.GetSceneObjectAssetUuids(UUID sceneObjectUuid, IDictionary`2 assetUuids) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\Framework\Scenes\UuidGatherer.cs:line 300
   at OpenSim.Region.Framework.Scenes.UuidGatherer.GatherAssetUuids(UUID assetUuid, AssetType assetType, IDictionary`2 assetUuids) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\Framework\Scenes\UuidGatherer.cs:line 119
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiveWriteRequest.SaveInvItem(InventoryItemBase inventoryItem, String path, Dictionary`2 options, IUserAccountService userAccountService) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiveWriteRequest.cs:line 166
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiveWriteRequest.SaveInvFolder(InventoryFolderBase inventoryFolder, String path, Boolean saveThisFolderItself, Dictionary`2 options, IUserAccountService userAccountService) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiveWriteRequest.cs:line 202
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiveWriteRequest.SaveInvFolder(InventoryFolderBase inventoryFolder, String path, Boolean saveThisFolderItself, Dictionary`2 options, IUserAccountService userAccountService) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiveWriteRequest.cs:line 197
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiveWriteRequest.Execute(Dictionary`2 options, IUserAccountService userAccountService) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiveWriteRequest.cs:line 322
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiverModule.ArchiveInventory(Guid id, String firstName, String lastName, String invPath, String pass, String savePath, Dictionary`2 options) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiverModule.cs:line 220
   at OpenSim.Region.CoreModules.Avatar.Inventory.Archiver.InventoryArchiverModule.HandleSaveInvConsoleCommand(String module, String[] cmdparams) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\CoreModules\Avatar\Inventory\Archiver\InventoryArchiverModule.cs:line 432
   at OpenSim.Framework.Console.Commands.Resolve(String[] cmd) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Framework\Console\CommandConsole.cs:line 373
   at OpenSim.Framework.Console.LocalConsole.ReadLine(String p, Boolean isCommand, Boolean e) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Framework\Console\LocalConsole.cs:line 480
   at OpenSim.Framework.Console.CommandConsole.Prompt() in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Framework\Console\CommandConsole.cs:line 592
   at OpenSim.Application.Main(String[] args) in c:\Documents and Settings\<username>\Desktop\OpenSim\OS 0.7.2 Dev\OpenSim\Region\Application\Application.cs:line 279
TagsNo tags attached.
Git Revision or version numberr/16513
Run Mode Standalone (Multiple Regions)
Physics EngineODE
Environment.NET / Windows32, .NET / Windows64
Mono VersionNone
ViewerAll
Attached Filestxt file icon config.txt [^] (13,707 bytes) 2011-08-17 01:26 [Show Content]

- Relationships

-  Notes
(0019611)
justincc (administrator)
2011-08-16 13:21

Are you sure that MemoryCacheEnabled = false in your FlotsamCache.ini? If so, please paste the results of "config show" or "config save".

Also, how much memory is on the machine that you're using, and what memory usage is show when you execute "show stats" at different points (e.g. before loading an iar/oar, afterwards, etc.)
(0019615)
mewtwo0641 (reporter)
2011-08-17 01:26

In my FlotsamCache.ini I have MemoryCacheEnabled = false and FileCacheEnabled = true. On one system Running 3 GB of RAM with 8 GB page file set up on my system and on my other system 4 GB of RAM with 8 GB page file.

Test before an Avatar logs in
------------------------------
show stats command:
Memory usage before IAR save is 225 MB
Memory usage after IAR save is 277 MB

task manager:
Opensim.exe Mem Usage: Appx 996 MB, Virtual Memory Size: Appx 1 GB


Test after an Avatar logs in:
-----------------------------

show stats command:
Memory usage before IAR save is 254 MB
Memory usage after IAR save is 246 MB

task manager:
Opensim.exe Mem Usage: Appx 523 MB, Virtual Memory Size: Appx 1 GB

See attached file for config save results please.
(0019778)
aiaustin (developer)
2011-08-29 01:01
edited on: 2011-08-29 01:09

I am seeing this too on one of the Openvue region servers that is running r/16521 that has been running for some time.. perhaps over a week. I am backing up oar files and saving the terrain for each region.

Just restarting the OpenSim.exe allowed the regions to all be saved as OARs successfuly without the out of memory errors.

Note also typo on error line text.. "since asset requster" - should be "requester"

I will update to latest 0.7.2 dev master, which the core Openvue ROBUST and central region servers is on, as I believe a memory leak was fixed since r/16521.

2011-08-29 08:57:16,466 ERROR - OpenSim.Region.CoreModules.World.Archiver.AssetsRequest [ARCHIVER]: Terminating archive creation since asset requster callback failed with System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

   at System.IO.MemoryStream.set_Capacity(Int32 value)

   at System.IO.MemoryStream.EnsureCapacity(Int32 value)

   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)

   at System.IO.BinaryWriter.Write(Single value)

   at OpenSim.Region.CoreModules.World.Terrain.FileLoaders.RAW32.SaveStream(Stream s, ITerrainChannel map) in d:\Temp\OpenSim\Region\CoreModules\World\Terrain\FileLoaders\RAW32.cs:line 156

   at OpenSim.Region.CoreModules.World.Terrain.TerrainModule.SaveToStream(String filename, Stream stream) in d:\Temp\OpenSim\Region\CoreModules\World\Terrain\TerrainModule.cs:line 364

   at OpenSim.Region.CoreModules.World.Archiver.ArchiveWriteRequestExecution.Save(ICollection`1 assetsFoundUuids, ICollection`1 assetsNotFoundUuids) in d:\Temp\OpenSim\Region\CoreModules\World\Archiver\ArchiveWriteRequestExecution.cs:line 134

   at OpenSim.Region.CoreModules.World.Archiver.ArchiveWriteRequestExecution.ReceivedAllAssets(ICollection`1 assetsFoundUuids, ICollection`1 assetsNotFoundUuids) in d:\Temp\OpenSim\Region\CoreModules\World\Archiver\ArchiveWriteRequestExecution.cs:line 88

   at OpenSim.Region.CoreModules.World.Archiver.AssetsRequest.PerformAssetsRequestCallback(Object o) in d:\Temp\OpenSim\Region\CoreModules\World\Archiver\AssetsRequest.cs:line 302

2011-08-29 08:57:16,535 DEBUG - OpenSim.Region.CoreModules.World.Archiver.AssetsRequest [ARCHIVER]: Successfully added 10 assets (0 assets notified missing)

2011-08-29 08:57:16,538 INFO - OpenSim.Region.CoreModules.World.Archiver.ArchiveWriteRequestExecution [ARCHIVER]: Added region settings to archive.

2011-08-29 08:57:16,541 INFO - OpenSim.Region.CoreModules.World.Archiver.ArchiveWriteRequestExecution [ARCHIVER]: Added parcel settings to archive.

2011-08-29 08:57:16,561 INFO - OpenSim.Region.CoreModules.World.Archiver.ArchiveWriteRequestExecution [ARCHIVER]: Added terrain information to archive.

2011-08-29 08:57:16,807 ERROR - OpenSim.Region.CoreModules.World.Archiver.AssetsRequest [ARCHIVER]: Terminating archive creation since asset requster callback failed with System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

   at System.IO.MemoryStream.set_Capacity(Int32 value)

   at System.IO.MemoryStream.EnsureCapacity(Int32 value)

   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)

   at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)

   at System.IO.StreamWriter.Write(String value)

   at System.Xml.XmlTextEncoder.Write(String text)

(0033846)
mewtwo0641 (reporter)
2019-01-19 04:36

Haven't seen this issue for a very long time; fixed by master
(0034313)
BillBlight (developer)
2019-02-05 12:27

Old Issues, closed, can be reopened if they still exist

- Issue History
Date Modified Username Field Change
2011-08-14 02:08 mewtwo0641 New Issue
2011-08-14 02:08 mewtwo0641 Git Revision => r/16513
2011-08-14 02:08 mewtwo0641 Run Mode => Standalone (Multiple Regions)
2011-08-14 02:08 mewtwo0641 Physics Engine => ODE
2011-08-14 02:08 mewtwo0641 Environment => .NET / Windows32, .NET / Windows64
2011-08-14 02:08 mewtwo0641 Mono Version => None
2011-08-14 02:08 mewtwo0641 Viewer => All
2011-08-14 14:35 mewtwo0641 Description Updated
2011-08-15 03:09 mewtwo0641 Description Updated
2011-08-15 03:11 mewtwo0641 Description Updated
2011-08-15 03:12 mewtwo0641 Description Updated
2011-08-16 13:21 justincc Note Added: 0019611
2011-08-16 13:21 justincc Status new => feedback
2011-08-17 01:26 mewtwo0641 Note Added: 0019615
2011-08-17 01:26 mewtwo0641 File Added: config.txt
2011-08-29 01:01 aiaustin Note Added: 0019778
2011-08-29 01:03 aiaustin Note Edited: 0019778
2011-08-29 01:09 aiaustin Note Edited: 0019778
2011-08-29 01:09 aiaustin Note Edited: 0019778
2019-01-19 04:36 mewtwo0641 Note Added: 0033846
2019-01-19 04:36 mewtwo0641 Status feedback => new
2019-01-19 04:36 mewtwo0641 Status new => resolved
2019-01-19 04:36 mewtwo0641 Fixed in Version => master (dev code)
2019-01-19 04:36 mewtwo0641 Resolution open => fixed
2019-01-19 04:36 mewtwo0641 Assigned To => mewtwo0641
2019-02-05 12:27 BillBlight Note Added: 0034313
2019-02-05 12:27 BillBlight Status resolved => closed


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker