Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008593opensim[GRID] Asset Servicepublic2019-10-06 09:392019-10-06 09:39
Assigned To 
PlatformLinuxOSUbuntuOS Version18.04
Product Version0.9.0.1 
Target VersionFixed in Version 
Summary0008593: Running an fsassets server in a second process leads to instability in both the grid and asset server. crashing
DescriptionI split out the FSAssets server into a second process, leaving the asset server in the main Grid process in place on the internal port for that server. So an asset server on 8003 and another asset only Robust instance on 8004. Both ports protected via firewall rules so not externally exposed.

Under load both the Grid server and asset server will exit (I have systemd doing automatic restarts). Caught this stack trace in the asset server and probably will find the same in the grid server.

23:24:18 - [BASE HTTP SERVER]: HandleRequest() threw exception System.IO.IOException: Sharing violation on path /var/opt/opensim/data/fsassets/tmp/spool/015B7A0ADADCF4C481F968639125122C8DA38921F4E5874DF4546930E2DA5AFD.asset
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) <0x7f9a416a3bc0 + 0x003e9> in <285579f54af44a2ca048dad6be20e190>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize) <0x7f9a416a35c0 + 0x00072> in <285579f54af44a2ca048dad6be20e190>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int)
  at System.IO.File.Create (System.String path, System.Int32 bufferSize) <0x7f9a23:25:02 - [FSASSETS]: Read stats: 48 files, 5356 ticks, avg 111.58, missing 437, FS 0ystem.IO.File.Create (System.String path) <0x7f9a41689360 + 0x00023> in <2R.O.B.U.S.T.# ca048dad6be20e190>:0

It would appear there is contention for temp files and likely assets failing to be created in these cases that also eventually causes a reset to happen.
Steps To ReproduceAs described above 2 instances of FSAssets, one in the grid server and one in a separate process.
Additional InformationI'd really love to move all the asset traffic out of the main grid process and there probably is a way to do that but the documentation on how to configure the connectors is pretty much non-existant. I suspect that the asset traffic in the Grid server is coming from the HGAssets activity while local grid assets are being served from my split out instance. Its likely I could configure a real single asset server given the docs to do it. But this problem remains and honestly I'd prefer redundant services that are load balanced so this will continue to be a problem.
TagsNo tags attached.
Git Revision or version numberUp to 39aa27af2b1294e313a53d33b4039e526643e75e
Run Mode Grid (1 Region per Sim)
Physics EngineubODE
EnvironmentMono / Linux64
Mono VersionOther
ViewerFirestorm 6.0.1
Attached Files

- Relationships

-  Notes
There are no notes attached to this issue.

- Issue History
Date Modified Username Field Change
2019-10-06 09:39 mike.dickson New Issue

Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker