|Anonymous | Login | Signup for a new account||2020-02-27 21:52 PST|
|Main | My View | View Issues | Change Log | Roadmap | Summary | My Account|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0008593||opensim||[GRID] Asset Service||public||2019-10-06 09:39||2019-10-22 04:01|
|Target Version||Fixed in Version|
|Summary||0008593: Running an fsassets server in a second process leads to instability in both the grid and asset server. crashing|
|Description||I split out the FSAssets server into a second process, leaving the asset server in the main Grid process in place on the internal port for that server. So an asset server on 8003 and another asset only Robust instance on 8004. Both ports protected via firewall rules so not externally exposed. |
Under load both the Grid server and asset server will exit (I have systemd doing automatic restarts). Caught this stack trace in the asset server and probably will find the same in the grid server.
23:24:18 - [BASE HTTP SERVER]: HandleRequest() threw exception System.IO.IOException: Sharing violation on path /var/opt/opensim/data/fsassets/tmp/spool/015B7A0ADADCF4C481F968639125122C8DA38921F4E5874DF4546930E2DA5AFD.asset
at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) <0x7f9a416a3bc0 + 0x003e9> in <285579f54af44a2ca048dad6be20e190>:0
at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize) <0x7f9a416a35c0 + 0x00072> in <285579f54af44a2ca048dad6be20e190>:0
at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int)
at System.IO.File.Create (System.String path, System.Int32 bufferSize) <0x7f9a23:25:02 - [FSASSETS]: Read stats: 48 files, 5356 ticks, avg 111.58, missing 437, FS 0ystem.IO.File.Create (System.String path) <0x7f9a41689360 + 0x00023> in <2R.O.B.U.S.T.# ca048dad6be20e190>:0
It would appear there is contention for temp files and likely assets failing to be created in these cases that also eventually causes a reset to happen.
|Steps To Reproduce||As described above 2 instances of FSAssets, one in the grid server and one in a separate process.|
|Additional Information||I'd really love to move all the asset traffic out of the main grid process and there probably is a way to do that but the documentation on how to configure the connectors is pretty much non-existant. I suspect that the asset traffic in the Grid server is coming from the HGAssets activity while local grid assets are being served from my split out instance. Its likely I could configure a real single asset server given the docs to do it. But this problem remains and honestly I'd prefer redundant services that are load balanced so this will continue to be a problem.|
|Tags||No tags attached.|
|Git Revision or version number||Up to 39aa27af2b1294e313a53d33b4039e526643e75e|
|Run Mode||Grid (1 Region per Sim)|
|Environment||Mono / Linux64|
|Increase ulimits, create own partition for assets, locate them in /srv, all options to reduce load for assets. If you truly want to remove load from robust due to assets go for SRAS instead.|
edited on: 2019-10-22 04:02
I am 110% sure the major bottleneck for FSASSETS is NOT the Linux file system. That said I appreciate the feedback. It still remains that there is a concurrency bug with asset creation that prohibits running more than one process over the same tree. One might be tempted to do a clustered filesystem and share the load across servers but I expect that would be equally problematic unless the underlying filesystem does full locking. Even then not sure you wouldn't have issues since it seems to be an access violation thats the main issue.
Also the reason I want to run multiple processes is not for throughput. That'd be nice but not my main issue. I am going for redundancy. I want to be able to take a process down for maintenance and have a redundant copy still serving requests.
|2019-10-06 09:39||mike.dickson||New Issue|
|2019-10-22 01:21||tampa||Note Added: 0035767|
|2019-10-22 04:01||mike.dickson||Note Added: 0035771|
|2019-10-22 04:02||mike.dickson||Note Edited: 0035771||View Revisions|
|Copyright © 2000 - 2012 MantisBT Group|