0008593: Running an fsassets server in a second process leads to instability in both the grid and asset server. crashing - MantisBT

Mantis Bug Tracker

View Issue Details [ Jump to Notes ]

[ Issue History ] [ Print ]

ID

Project

Category

View Status

Date Submitted

Last Update

0008593

opensim

[GRID] Asset Service

public

2019-10-06 09:39

2019-10-22 04:01

Reporter

mike.dickson

Assigned To

Priority

normal

Severity

crash

Reproducibility

always

Status

new

Resolution

open

Platform

Linux

OS

Ubuntu

OS Version

18.04

Product Version

0.9.0.1

Target Version

Fixed in Version

Summary

0008593: Running an fsassets server in a second process leads to instability in both the grid and asset server. crashing

Description

I split out the FSAssets server into a second process, leaving the asset server in the main Grid process in place on the internal port for that server. So an asset server on 8003 and another asset only Robust instance on 8004. Both ports protected via firewall rules so not externally exposed.

Under load both the Grid server and asset server will exit (I have systemd doing automatic restarts). Caught this stack trace in the asset server and probably will find the same in the grid server.

23:24:18 - [BASE HTTP SERVER]: HandleRequest() threw exception System.IO.IOException: Sharing violation on path /var/opt/opensim/data/fsassets/tmp/spool/015B7A0ADADCF4C481F968639125122C8DA38921F4E5874DF4546930E2DA5AFD.asset
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) <0x7f9a416a3bc0 + 0x003e9> in <285579f54af44a2ca048dad6be20e190>:0
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize) <0x7f9a416a35c0 + 0x00072> in <285579f54af44a2ca048dad6be20e190>:0
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare,int)
  at System.IO.File.Create (System.String path, System.Int32 bufferSize) <0x7f9a23:25:02 - [FSASSETS]: Read stats: 48 files, 5356 ticks, avg 111.58, missing 437, FS 0ystem.IO.File.Create (System.String path) <0x7f9a41689360 + 0x00023> in <2R.O.B.U.S.T.# ca048dad6be20e190>:0

It would appear there is contention for temp files and likely assets failing to be created in these cases that also eventually causes a reset to happen.

Steps To Reproduce

As described above 2 instances of FSAssets, one in the grid server and one in a separate process.

Additional Information

I'd really love to move all the asset traffic out of the main grid process and there probably is a way to do that but the documentation on how to configure the connectors is pretty much non-existant. I suspect that the asset traffic in the Grid server is coming from the HGAssets activity while local grid assets are being served from my split out instance. Its likely I could configure a real single asset server given the docs to do it. But this problem remains and honestly I'd prefer redundant services that are load balanced so this will continue to be a problem.

Tags

No tags attached.

Git Revision or version number

Up to 39aa27af2b1294e313a53d33b4039e526643e75e

Run Mode

Grid (1 Region per Sim)

Physics Engine

ubODE

Script Engine

Environment

Mono / Linux64

Mono Version

Other

Viewer

Firestorm 6.0.1

Relationships

Notes
(0035767) tampa (reporter) 2019-10-22 01:21	Increase ulimits, create own partition for assets, locate them in /srv, all options to reduce load for assets. If you truly want to remove load from robust due to assets go for SRAS instead.

(0035771) mike.dickson (reporter) 2019-10-22 04:01 edited on: 2019-10-22 04:02	I am 110% sure the major bottleneck for FSASSETS is NOT the Linux file system. That said I appreciate the feedback. It still remains that there is a concurrency bug with asset creation that prohibits running more than one process over the same tree. One might be tempted to do a clustered filesystem and share the load across servers but I expect that would be equally problematic unless the underlying filesystem does full locking. Even then not sure you wouldn't have issues since it seems to be an access violation thats the main issue. Also the reason I want to run multiple processes is not for throughput. That'd be nice but not my main issue. I am going for redundancy. I want to be able to take a process down for maintenance and have a redundant copy still serving requests.

Issue History
Date Modified	Username	Field	Change
2019-10-06 09:39	mike.dickson	New Issue
2019-10-22 01:21	tampa	Note Added: 0035767
2019-10-22 04:01	mike.dickson	Note Added: 0035771
2019-10-22 04:02	mike.dickson	Note Edited: 0035771	View Revisions

Copyright © 2000 - 2012 MantisBT Group