0008593: Running an fsassets server in a second process leads to instability in both the grid and asset server. crashing

View Issue Details [ Jump to Notes ]

[ Issue History ] [ Print ]

Project

Category

View Status

Date Submitted

Last Update

0008593

opensim

[GRID] Asset Service

public

2019-10-06 09:39

Reporter

mike.dickson

Assigned To

Priority

normal

Severity

crash

Reproducibility

always

Status

new

Resolution

open

Platform

Linux

Ubuntu

OS Version

18.04

Product Version

0.9.0.1

Target Version

Fixed in Version

Summary

0008593: Running an fsassets server in a second process leads to instability in both the grid and asset server. crashing

Description

I split out the FSAssets server into a second process, leaving the asset server in the main Grid process in place on the internal port for that server. So an asset server on 8003 and another asset only Robust instance on 8004. Both ports protected via firewall rules so not externally exposed.

Under load both the Grid server and asset server will exit (I have systemd doing automatic restarts). Caught this stack trace in the asset server and probably will find the same in the grid server.

23:24:18 - [BASE HTTP SERVER]: HandleRequest() threw exception System.IO.IOException: Sharing violation on path /var/opt/opensim/data/fsassets/tmp/spool/015B7A0ADADCF4C481F968639125122C8DA38921F4E5874DF4546930E2DA5AFD.asset
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) <0x7f9a416a3bc0 + 0x003e9> in <285579f54af44a2ca048dad6be20e190>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize) <0x7f9a416a35c0 + 0x00072> in <285579f54af44a2ca048dad6be20e190>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int)
  at System.IO.File.Create (System.String path, System.Int32 bufferSize) <0x7f9a23:25:02 - [FSASSETS]: Read stats: 48 files, 5356 ticks, avg 111.58, missing 437, FS 0ystem.IO.File.Create (System.String path) <0x7f9a41689360 + 0x00023> in <2R.O.B.U.S.T.# ca048dad6be20e190>:0

It would appear there is contention for temp files and likely assets failing to be created in these cases that also eventually causes a reset to happen.

Steps To Reproduce

As described above 2 instances of FSAssets, one in the grid server and one in a separate process.

Additional Information

I'd really love to move all the asset traffic out of the main grid process and there probably is a way to do that but the documentation on how to configure the connectors is pretty much non-existant. I suspect that the asset traffic in the Grid server is coming from the HGAssets activity while local grid assets are being served from my split out instance. Its likely I could configure a real single asset server given the docs to do it. But this problem remains and honestly I'd prefer redundant services that are load balanced so this will continue to be a problem.

Issue History
Date Modified	Username	Field	Change
2019-10-06 09:39	mike.dickson	New Issue

Relationships

Notes
There are no notes attached to this issue.