Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008143opensim[GRID] Asset Servicepublic2017-04-04 14:462017-05-27 14:49
Reportermewtwo0641 
Assigned ToUbitUmarov 
PrioritynormalSeveritymajorReproducibilityalways
StatusresolvedResolutionfixed 
PlatformOSOS Version
Product Versionmaster (dev code) 
Target VersionFixed in Version 
Summary0008143: Mesh (and possibly texture) asset load is noticeably slower than from OS 0.8.2
DescriptionSince commit d1810866b3177c9c6026f619426c1bda72e14374

Meshes (and possibly textures) take a good noticeable bit longer to download their assets and rez in the viewer than they did from OS 0.8.2.

On OpenSim Versions <= 0.8.2, mesh objects would more or less instantly load for me. On master code, the same objects can take anywhere from a few seconds to a few minutes depending on how large the object is.

It isn't so bad when there's only a few items in the region to rez... But it doesn't scale up well at all when one wants to build a whole region... The more that's on the region the slower it is for these objects to load.

I understand the immediate reaction to the above statement would be that "Well of course! There's more there for the server to send and for the viewer to download. This takes more time." That is true but... This doesn't entirely seem to be the case for this particular issue. There is a very noticeable performance drop between testing the region on 0.8.2 and testing the same region on master code. (See Steps to Reproduce)

As a side note, I don't know if this would have anything to do with it, but I did try various physics engines to see if any one in particular would help the issue but none of them seem to affect it one way or the other. Tested ODE, ubODE, and BulletSim.
Steps To ReproduceRez some mesh objects or go to a region running a recent build of OS with mesh objects (Region method is preferable since it gives a better idea of the larger scale performance drop).

I've been able to verify the issue by testing my region (fairly densely populated region) in this manner:

1. Test on OS 0.8.2: All objects are loaded within a few minutes.

2. Test on OS 0.9.0 Master: A bunch of objects are still not loaded even a half hour (or longer in some cases) later

3. Reverted the changes at commit d1810866, recompiled, and tried the same region again: All objects are loaded within a few minutes. Reverting this commit also seems to help with slow texture rez as well.

Between each test I am making sure to run fcache clear, clear the MeshCache folder, and clear my viewer cache to be sure that I'm testing the network performance and not getting these assets from cache.
Additional InformationGit Bisect Results:

$ git bisect good
d1810866b3177c9c6026f619426c1bda72e14374 is the first bad commit
commit d1810866b3177c9c6026f619426c1bda72e14374
Author: UbitUmarov <ajlduarte@sapo.pt>
Date: Sat Jul 16 19:53:41 2016 +0100

    simplify http textures and meshs Throttles

:040000 040000 192ba9e8af7e3f48f42ecc7796754e8ad061e17b 2f68294f5fd4a1f47a5095b68db301c5e1ec3757 M OpenSim
TagsNo tags attached.
Git Revision or version number
Run Mode Standalone (Multiple Regions)
Physics EngineubODE
EnvironmentUnknown
Mono VersionNone
ViewerN/A
Attached Files

- Relationships

-  Notes
(0031705)
UbitUmarov (administrator)
2017-04-04 16:07

- 0.8.2 did not have any http throttles
- 0.9 introduced avn throttles code
- that commit removed a exception processing that did them not to limit the bandwidth in many cases.
- does not depend on physics engine

so you are very right, http assets data fetch is slower per user on 0.9

throttles are a necessary evil. Without them a single user can take all the available bandwidth. That user will get assets very fast, but others need to wait.
They also matter in the case where there is a single user in region. http packets have priority over UDP, so receiving assets via http can block UDP, even causing packets drop, either on the region or the viewer machine, and for us UDP should have higher priority, at least in current protocol.
Also other http requests need to have higher priority than assets data fetch, so so currently do not have throttles.
Singularity viewer added viewer side throttles for http, limiting those assets requests to prevent viewer side problems, even on grids that use fast and cached http services.
Other viewers don't and Singularity does not provide the bandwidth setting so we use the UDP throttles.

we do agree that this does need a lot of improvement, but we do need to change the http server, and somehow integrate udp and http control with better awareness of overall load.
(0031706)
mewtwo0641 (reporter)
2017-04-04 16:47
edited on: 2017-04-04 16:48

Ahh I see. I do agree that letting one user suck up all available bandwidth is a bad thing; but are there currently any or will there be any future plans to allow for server side configuration in OpenSim.ini of these new throttles on a per user basis? As it stands now... It can be painfully slow on a modestly populated region even for just one avatar.

(0031707)
UbitUmarov (administrator)
2017-04-04 17:06

Yes I understand.
I did start some changes on http on a branch called httptests, but didn't even reached the throttles problem. for now it just does http pipeline a bit better. That branch is now a just bit outdated, I will update soon.

You can try increasing the bandwidth settings in viewer, unfortunately viewers may also get in trouble with 2 much UDP traffic.

Reduced view range also helps, most time you are not looking to the entire region, region culling option may also help a bit (but still experimental, and not sure as I type, if I added to master or just httptests)
(0031708)
mewtwo0641 (reporter)
2017-04-04 17:47

Most of the time I usually keep my view distance down pretty small (Around 96m) for day to day use and only increasing it when I need to do some long distance editing of things across my region(s). I also usually keep my bandwidth slider in the viewer's network settings at max but have never really encountered any glaring problems with that until the appearance of this issue; but lowering it also doesn't seem to have any effect on the issue.

I have not yet tried region culling; Where might I find that option?
(0031709)
UbitUmarov (administrator)
2017-04-04 18:24

its on opensim.ini.example section[InterestManagement], ypu can copy it to opensim.ini, same section
but don't expect miracles.. has problems on its own and also work in progress :)
(0031717)
mewtwo0641 (reporter)
2017-04-05 14:36

Thank you! I gave the region culling option a try today but it didn't seem to make a whole lot of difference. It might just be my imagination but turning this option on did seem to make the experience of walking around while things are loading a bit smoother maybe (i.e. Not as bad CPU/GPU lag)? But as far as network performance; Mesh and textures are still quite slow to download.
(0031718)
UbitUmarov (administrator)
2017-04-05 14:47

with that option region only sends to viewer prims in view range, so less objects on viewer, less memory, better performance.
But as you walk way from original point, a new set with prims that did enter view range is sent, and viewer is told to delete the ones that left view range.
this happens every x (30??) meters
this updates can cause load spikes if the number of prim changes is very high.

if you don't have textures and mesh in cache, well doesn't help much
(0031719)
mewtwo0641 (reporter)
2017-04-05 15:21
edited on: 2017-04-05 15:26

Yes, I understand the cache is there to speed things along, but the reason I am clearing cache for this issue is so that I'm sure that I'm testing the network side of things and not my local disk access speed :)

Also... I'm not sure if it is related to the issue, but on my setup, I receive an http server error message on a regular basis. It isn't spamming the console over and over like just a constant torrent of messages but it pops up say every few minutes or so.

2017-03-11 20:40:35,282 WARN (STP:PoolService:33) - OpenSim.Framework.Servers.HttpServer.PollServiceHttpRequest [POLL SERVICE WORKER THREAD]: Error
System.Net.Sockets.SocketException (0x80004005): An established connection was aborted by the software in your host machine
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at HttpServer.HttpClientContext.Send(Byte[] buffer, Int32 offset, Int32 size) in C:\opensim\opensim-libs\HttpServer\trunk\HttpServer\HttpClientContext.cs:line 520
   at HttpServer.HttpResponse.Send() in C:\opensim\opensim-libs\HttpServer\trunk\HttpServer\HttpResponse.cs:line 280
   at OpenSim.Framework.Servers.HttpServer.OSHttpResponse.Send() in e:\opensim\OpenSim\Framework\Servers\HttpServer\OSHttpResponse.cs:line 325
   at OpenSim.Framework.Servers.HttpServer.PollServiceHttpRequest.DoHTTPGruntWork(BaseHttpServer server, Hashtable responsedata) in e:\opensim\OpenSim\Framework\Servers\HttpServer\PollServiceHttpRequest.cs:line 94

(0031720)
UbitUmarov (administrator)
2017-04-05 16:07

it is a debug message that is just annoying, harmless and unrelated to current issue

- Issue History
Date Modified Username Field Change
2017-04-04 14:46 mewtwo0641 New Issue
2017-04-04 14:52 mewtwo0641 Description Updated View Revisions
2017-04-04 15:00 mewtwo0641 Steps to Reproduce Updated View Revisions
2017-04-04 16:07 UbitUmarov Note Added: 0031705
2017-04-04 16:47 mewtwo0641 Note Added: 0031706
2017-04-04 16:48 mewtwo0641 Note Edited: 0031706 View Revisions
2017-04-04 17:06 UbitUmarov Note Added: 0031707
2017-04-04 17:47 mewtwo0641 Note Added: 0031708
2017-04-04 18:24 UbitUmarov Note Added: 0031709
2017-04-05 14:36 mewtwo0641 Note Added: 0031717
2017-04-05 14:47 UbitUmarov Note Added: 0031718
2017-04-05 15:21 mewtwo0641 Note Added: 0031719
2017-04-05 15:26 mewtwo0641 Note Edited: 0031719 View Revisions
2017-04-05 16:07 UbitUmarov Note Added: 0031720
2017-05-27 14:49 UbitUmarov Status new => resolved
2017-05-27 14:49 UbitUmarov Resolution open => fixed
2017-05-27 14:49 UbitUmarov Assigned To => UbitUmarov


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker