MantisBT - opensim
View Issue Details
0008696opensim[REGION] OpenSim Corepublic2020-05-01 00:102020-05-05 05:47
resolvedno change required 
Grid (Multiple Regions per Sim)
.NET / Windows64
0008696: Very frequent timeouts with SynchronousRestObjectRequester causing sim hang ups and slowness
I am seeing a lot of timeouts related to SynchronousRestObjectRequester on master lately when loading OS, uploading assets, loading OARs/IARs, etc. that were not previously a problem.

At first glance the timeouts seem to be causing operations related to them to take an extremely long time to finish; for example, an OAR file of a project that I am working on for someone took almost 20 minutes to load where normally, before this issue came up, it'd only take a minute tops. I also have no idea if the assets that were meant to be uploaded from the file were successfully uploaded or not.

Example WebException Message:

01:57:29 - [SynchronousRestObjectRequester]: WebException for GET [^] OpenSim.Framework.AssetBase The operation has timed out
No tags attached.
Issue History
2020-05-01 00:10mewtwo0641New Issue
2020-05-01 00:10mewtwo0641SummaryVery frequent timeouts with SynchronousRestObjectRequester => Very frequent timeouts with SynchronousRestObjectRequester causing sim hang ups and slowness
2020-05-01 00:21tampaNote Added: 0036414
2020-05-01 03:58mewtwo0641Note Added: 0036415
2020-05-02 13:11tampaNote Added: 0036422
2020-05-04 04:23mewtwo0641Note Added: 0036442
2020-05-04 04:33tampaNote Added: 0036443
2020-05-05 05:47mewtwo0641Note Added: 0036444
2020-05-05 05:47mewtwo0641Statusnew => resolved
2020-05-05 05:47mewtwo0641Resolutionopen => no change required
2020-05-05 05:47mewtwo0641Assigned To => mewtwo0641

2020-05-01 00:21   
Do you have a longer log with more of these happening?
2020-05-01 03:58   
I do not at the moment. I will have to go back later and see if I can trigger it again and grab some more extensive logs.

An observation that I have made though is, I can't get this to trigger on my other system... The primary difference between the two is that one OS installation + database is on an NVMe drive and the other is on a standard HDD. Both are local to the drives and both are being accessed/logged into over LAN, nothing is being done over WAN/Internet, and both are on commit 5fa81a (master as of this moment).
2020-05-02 13:11   
If this is happening on the system with the HDDs then it points toward my initial thought and that is simply database overload. It takes time to get the data out the DB and prep it for sending, that may be too much for the timeout. A good way to reduce the problems with that is tuning the database configuration. There are a few variables that can be set there that will use more system resource to cache data longer and hold more data in memory. This can help, but ultimately it is becoming more and more evident that OpenSim does need fast I/O in order to really run properly, HDDs are just not capable of handling loads of small requests all the time.
2020-05-04 04:23   
I did some more testing today and I think that you are correct with the HDD being a cause of the issue. Just to test, I tried migrating the assets off of MySQL onto FS Assets and encountered quite a few timeouts relating to MySQL not responding while it did so (Including a ROBUST crash related to trying to access a file while it was in use for some odd reason). I had to restart the migration about 8 times just to get it all in there. Now that I have all the assets being served off FS Assets I seem to be encountering far fewer issues... Although I suspect that it may only have delayed having the issue again sometime in the future as the database grows.

The database is actually not -that- big at around 6 GB (comparatively speaking to FS Asset's recommendation of being used if the database is expected to be >50 GB)... Which is why I didn't consider moving to FS Assets previously.

This is all being done on a test DB of course.
2020-05-04 04:33   
There are some settings to tweak from the standard database install and ideally you want region stuff and grid stuff on their own systems. Tweaking the database is different from system to system so best to just read some guides on how to do it and apply it according to the variables you have.

I use a similar file-based asset system, but on the 3 million assets on there, god knows how much space that and its db takes I don't see it requesting assets more than once. If the hardware underneath is decent and some tweaks are applied you can generally grow this out quite large without much trouble.
2020-05-05 05:47   
I am closing this mantis because it is looking like an issue on my end rather than an issue with OpenSim. Thank you very much for your tips and suggestions :)