Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008593opensim[GRID] Asset Servicepublic2019-10-06 09:392019-10-22 04:01
Assigned To 
PlatformLinuxOSUbuntuOS Version18.04
Product Version0.9.0.1 
Target VersionFixed in Version 
Summary0008593: Running an fsassets server in a second process leads to instability in both the grid and asset server. crashing
DescriptionI split out the FSAssets server into a second process, leaving the asset server in the main Grid process in place on the internal port for that server. So an asset server on 8003 and another asset only Robust instance on 8004. Both ports protected via firewall rules so not externally exposed.

Under load both the Grid server and asset server will exit (I have systemd doing automatic restarts). Caught this stack trace in the asset server and probably will find the same in the grid server.

23:24:18 - [BASE HTTP SERVER]: HandleRequest() threw exception System.IO.IOException: Sharing violation on path /var/opt/opensim/data/fsassets/tmp/spool/015B7A0ADADCF4C481F968639125122C8DA38921F4E5874DF4546930E2DA5AFD.asset
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) <0x7f9a416a3bc0 + 0x003e9> in <285579f54af44a2ca048dad6be20e190>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize) <0x7f9a416a35c0 + 0x00072> in <285579f54af44a2ca048dad6be20e190>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int)
  at System.IO.File.Create (System.String path, System.Int32 bufferSize) <0x7f9a23:25:02 - [FSASSETS]: Read stats: 48 files, 5356 ticks, avg 111.58, missing 437, FS 0ystem.IO.File.Create (System.String path) <0x7f9a41689360 + 0x00023> in <2R.O.B.U.S.T.# ca048dad6be20e190>:0

It would appear there is contention for temp files and likely assets failing to be created in these cases that also eventually causes a reset to happen.
Steps To ReproduceAs described above 2 instances of FSAssets, one in the grid server and one in a separate process.
Additional InformationI'd really love to move all the asset traffic out of the main grid process and there probably is a way to do that but the documentation on how to configure the connectors is pretty much non-existant. I suspect that the asset traffic in the Grid server is coming from the HGAssets activity while local grid assets are being served from my split out instance. Its likely I could configure a real single asset server given the docs to do it. But this problem remains and honestly I'd prefer redundant services that are load balanced so this will continue to be a problem.
TagsNo tags attached.
Git Revision or version numberUp to 39aa27af2b1294e313a53d33b4039e526643e75e
Run Mode Grid (1 Region per Sim)
Physics EngineubODE
Script Engine
EnvironmentMono / Linux64
Mono VersionOther
ViewerFirestorm 6.0.1
Attached Files

- Relationships

-  Notes
tampa (reporter)
2019-10-22 01:21

Increase ulimits, create own partition for assets, locate them in /srv, all options to reduce load for assets. If you truly want to remove load from robust due to assets go for SRAS instead.
mike.dickson (reporter)
2019-10-22 04:01
edited on: 2019-10-22 04:02

I am 110% sure the major bottleneck for FSASSETS is NOT the Linux file system. That said I appreciate the feedback. It still remains that there is a concurrency bug with asset creation that prohibits running more than one process over the same tree. One might be tempted to do a clustered filesystem and share the load across servers but I expect that would be equally problematic unless the underlying filesystem does full locking. Even then not sure you wouldn't have issues since it seems to be an access violation thats the main issue.

Also the reason I want to run multiple processes is not for throughput. That'd be nice but not my main issue. I am going for redundancy. I want to be able to take a process down for maintenance and have a redundant copy still serving requests.

- Issue History
Date Modified Username Field Change
2019-10-06 09:39 mike.dickson New Issue
2019-10-22 01:21 tampa Note Added: 0035767
2019-10-22 04:01 mike.dickson Note Added: 0035771
2019-10-22 04:02 mike.dickson Note Edited: 0035771 View Revisions

Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker