|Anonymous | Login | Signup for a new account||2020-07-02 08:31 PDT|
|Main | My View | View Issues | Change Log | Roadmap | Summary | My Account|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0008696||opensim||[REGION] OpenSim Core||public||2020-05-01 00:10||2020-05-05 05:47|
|Status||resolved||Resolution||no change required|
|Target Version||Fixed in Version|
|Summary||0008696: Very frequent timeouts with SynchronousRestObjectRequester causing sim hang ups and slowness|
|Description||I am seeing a lot of timeouts related to SynchronousRestObjectRequester on master lately when loading OS, uploading assets, loading OARs/IARs, etc. that were not previously a problem.|
At first glance the timeouts seem to be causing operations related to them to take an extremely long time to finish; for example, an OAR file of a project that I am working on for someone took almost 20 minutes to load where normally, before this issue came up, it'd only take a minute tops. I also have no idea if the assets that were meant to be uploaded from the file were successfully uploaded or not.
Example WebException Message:
01:57:29 - [SynchronousRestObjectRequester]: WebException for GET http://127.0.0.1:8003/assets/a3c80df9-8212-4e00-ea98-06fe05a7b510 [^] OpenSim.Framework.AssetBase The operation has timed out
|Tags||No tags attached.|
|Git Revision or version number|
|Run Mode||Grid (Multiple Regions per Sim)|
|Environment||.NET / Windows64|
|Do you have a longer log with more of these happening?|
I do not at the moment. I will have to go back later and see if I can trigger it again and grab some more extensive logs.
An observation that I have made though is, I can't get this to trigger on my other system... The primary difference between the two is that one OS installation + database is on an NVMe drive and the other is on a standard HDD. Both are local to the drives and both are being accessed/logged into over LAN, nothing is being done over WAN/Internet, and both are on commit 5fa81a (master as of this moment).
|If this is happening on the system with the HDDs then it points toward my initial thought and that is simply database overload. It takes time to get the data out the DB and prep it for sending, that may be too much for the timeout. A good way to reduce the problems with that is tuning the database configuration. There are a few variables that can be set there that will use more system resource to cache data longer and hold more data in memory. This can help, but ultimately it is becoming more and more evident that OpenSim does need fast I/O in order to really run properly, HDDs are just not capable of handling loads of small requests all the time.|
I did some more testing today and I think that you are correct with the HDD being a cause of the issue. Just to test, I tried migrating the assets off of MySQL onto FS Assets and encountered quite a few timeouts relating to MySQL not responding while it did so (Including a ROBUST crash related to trying to access a file while it was in use for some odd reason). I had to restart the migration about 8 times just to get it all in there. Now that I have all the assets being served off FS Assets I seem to be encountering far fewer issues... Although I suspect that it may only have delayed having the issue again sometime in the future as the database grows.
The database is actually not -that- big at around 6 GB (comparatively speaking to FS Asset's recommendation of being used if the database is expected to be >50 GB)... Which is why I didn't consider moving to FS Assets previously.
This is all being done on a test DB of course.
There are some settings to tweak from the standard database install and ideally you want region stuff and grid stuff on their own systems. Tweaking the database is different from system to system so best to just read some guides on how to do it and apply it according to the variables you have.
I use a similar file-based asset system, but on the 3 million assets on there, god knows how much space that and its db takes I don't see it requesting assets more than once. If the hardware underneath is decent and some tweaks are applied you can generally grow this out quite large without much trouble.
|I am closing this mantis because it is looking like an issue on my end rather than an issue with OpenSim. Thank you very much for your tips and suggestions :)|
|2020-05-01 00:10||mewtwo0641||New Issue|
|2020-05-01 00:10||mewtwo0641||Summary||Very frequent timeouts with SynchronousRestObjectRequester => Very frequent timeouts with SynchronousRestObjectRequester causing sim hang ups and slowness|
|2020-05-01 00:21||tampa||Note Added: 0036414|
|2020-05-01 03:58||mewtwo0641||Note Added: 0036415|
|2020-05-02 13:11||tampa||Note Added: 0036422|
|2020-05-04 04:23||mewtwo0641||Note Added: 0036442|
|2020-05-04 04:33||tampa||Note Added: 0036443|
|2020-05-05 05:47||mewtwo0641||Note Added: 0036444|
|2020-05-05 05:47||mewtwo0641||Status||new => resolved|
|2020-05-05 05:47||mewtwo0641||Resolution||open => no change required|
|2020-05-05 05:47||mewtwo0641||Assigned To||=> mewtwo0641|
|Copyright © 2000 - 2012 MantisBT Group|