Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008343opensim[GRID] Other Servicepublic2018-07-19 07:402018-08-13 20:44
Reportereform 
Assigned To 
PriorityhighSeveritymajorReproducibilityalways
StatusnewResolutionopen 
PlatformOSOS Version
Product Version0.9.0.1 
Target VersionFixed in Version 
Summary0008343: 0.9.x has severe (uncontrollable?) HTTP throttling compared to 0.8.x
DescriptionHello, I am running a sim which relies on a high volume of scripts using llHTTPRequest and osSetDynamicTextureData over localhost.

After installing and running 0.9.1. with (virtually) identical settings to 0.8.2.1 the new version seems to introduce tremendous sluggishness in the scripts downloading data from localhost over HTTP (including many dynamic textures loading only after a very very long delay). If I boot 0.8.2.1 running off the same database and assetcache, there is no sluggishness - everything works normally.

I also get a lot of this warning in the log exclusively with the new opensim version.

WARN [POLL SERVICE WORKER THREAD]: Error
System.Net.Sockets.SocketException (0x80004005): An established connection was aborted by the software in your host machine
   at HttpServer.HttpClientContext.Send(Byte[] buffer, Int32 offset, Int32 size) in C:\opensim\opensim-libs\HttpServer\trunk\HttpServer\HttpClientContext.cs:line 521
   at HttpServer.HttpResponse.Send() in C:\opensim\opensim-libs\HttpServer\trunk\HttpServer\HttpResponse.cs:line 280
   at OpenSim.Framework.Servers.HttpServer.PollServiceHttpRequest.DoHTTPGruntWork(BaseHttpServer server, Hashtable responsedata)

So the questions I have are: does 0.9.x deliberately throttle HTTP connections? If so, is there a setting that can disable it? Alternatively, which part of the source code is responsible for it?

If somebody could help me with a solution so my sim can work normally with 0.9.0.1 I would greatly appreciate it. Thank you.

This report may be related to my issue: http://opensimulator.org/mantis/view.php?id=8143 [^]
TagsNo tags attached.
Git Revision or version numberopensim-0.9.0.1 Release
Run Mode Standalone (Multiple Regions)
Physics EngineBulletSim
Environment.NET / Windows64
Mono VersionNone
ViewerFirestorm
Attached Files

- Relationships

-  Notes
(0032808)
paela argus (reporter)
2018-07-21 06:49

Hello yes is throttle the responsive
Code part are in:
OpenSim/Region/ClientStack/Linden/Caps
File name for mesh: GetMeshModule.cs
File name for Texture: GetTextureModule.cs

good luck.
(0032809)
BillBlight (developer)
2018-07-21 09:02

Yes yes it does .. And there are reasons for it, but if you don't mind everyone on the sim lagging while some one with a overly heavy avatar steals all the band width, or when a few heavy avatars steal all of it and lag the whole sim ..

Feel free to remove them ..


The choice of the throttles is to improve the overall experience of everyone on the sim not just the experience of one or two avatars.
(0032811)
BillBlight (developer)
2018-07-21 09:10
edited on: 2018-07-21 09:12

The other thing is this error ..

"WARN [POLL SERVICE WORKER THREAD]: Error
System.Net.Sockets.SocketException (0x80004005): An established connection was aborted by the software in your host machine
   at HttpServer.HttpClientContext.Send(Byte[] buffer, Int32 offset, Int32 size) in C:\opensim\opensim-libs\HttpServer\trunk\HttpServer\HttpClientContext.cs:line 521
   at HttpServer.HttpResponse.Send() in C:\opensim\opensim-libs\HttpServer\trunk\HttpServer\HttpResponse.cs:line 280
   at OpenSim.Framework.Servers.HttpServer.PollServiceHttpRequest.DoHTTPGruntWork(BaseHttpServer server, Hashtable responsedata)"

Is due to networking and mono config, the host machine is closing the connections prematurely , it is not a specific opensim error ..

I used to get this error until I tweaked my networking and mono variables, if I can find in my notes what I specifically changed I'll post it here, but has been a long time ago .. If you are running windows firewall, it could also cause this issue ..

(0032814)
paela argus (reporter)
2018-07-21 11:48

it improves nothing or almost this code it should find this option false / true in OpenSim.ini for a home connection I want to understand but on a server in datacenter this restriction is completely useless.
Its been 1 year see more, that everyone screams on osgrid loading times since this set up, it will have to really look into this problem and seriously.
(0032815)
BillBlight (developer)
2018-07-21 11:50

I'm not sure where you see the issue we had 30 avatars on my grid, for an event, and NO one complained about lag or slow loading, saw no errors on the console and everyone's mesh loaded fast , so again not going to argue that the throttles are there, they are...

But I am going to say that when things are properly setup they don't cause a "sky is falling" issue ..
(0032816)
BillBlight (developer)
2018-07-21 11:52

The other thing to note is currently I doubt that osgrid is using the robust code that has the http pipeline in it, so attached regions that are using it , don't have access to it on the asset server side.
(0032817)
BillBlight (developer)
2018-07-21 11:57

The other thing of note, is that if you are using Firestorm , they have the httpppipeline force disabled for opensim, even if you have it turned on ..
(0032818)
BillBlight (developer)
2018-07-21 12:05
edited on: 2018-07-21 12:08

I made an assumption that this mantis was based off of current master code and not actually 0.9.0.1 , so I could be wrong about that ..


So disregard my rant, if this is being applied to 0.9.0.1, but if it is being applied to master code, I stand firm..

(0032823)
eform (reporter)
2018-07-23 06:41
edited on: 2018-07-23 06:43

Thanks for your messages. I know removing the HTTP throttle would seem like a regression, but on my particular sim only administrators can script and script connections outside of localhost are blocked. An INI option to enable/disable this throttle would provide extra flexibility in my opinion.

I know that GetMeshModule and GetTextureModule have new throttle code in 0.9.x over 0.8.x (I'm using release versions by the way), but without knowing too much about Opensim's architecture, I can understand that GetTextureModule could have an effect on the osSetDynamicTextureData function, but could either of these files throttle use of llHTTPRequest?

It just doesn't seem intuitive because my sim's use of llHTTPRequest does not involve the transfers of meshes or textures, only text.

EDIT: I know the error listed in the report isn't actually an Opensim error, but it doesn't occur under 0.8.x so whatever 0.9.x is doing, it triggers it now for me.

(0032825)
BillBlight (developer)
2018-07-23 07:01

I use external http requests all the time, and they are not blocked??

As long as the two required settings are set in the [Network] section of the ini ..

ExternalHostNameForLSL=
OutboundDisallowForUserScripts = ""

I also set the other settings a bit higher ..

MaxRequestConcurrency = 60
HttpBodyMaxLenMAX=32768
(0032827)
tampa (reporter)
2018-07-23 07:18

Just throwing more scripts and httprequests at a region expecting things to work is a sure way of killing performance all around. Do you actually need that many httprequests and if you do, maybe there is a more clever way of handling all them instead of making loads of individual ones? I'm all for removing limits usually, but there is always the argument of protecting both performance and users from misusing, overusing and breaking the software because they fail to engage in efficient coding practices and avoid pitfalls such as these.

I'm with watcher64 there, there is a reason for these limitations and normally, with a bit of ingenuity and good code, you almost never encounter them. I have had my fair share of running into event time limits and other limits that I could all easily resolve with a better strategy toward handling large amounts of data. OpenSim, nor any other software, should be required to adjust to bad use cases or misuse and, again, I am usually the first to cry for removing limitations.
(0032830)
eform (reporter)
2018-07-24 01:04
edited on: 2018-07-24 01:05

Thanks again for your messages. Increasing MaxRequestConcurrency by a little off the default seemed to reduce the throttling issues my sim is experiencing to an extent, but many responsiveness issues remain. When I raised MaxRequestConcurrency too high, the entire sim became bogged down by latency, not just the scripts. If I had to make any other observation, it was that scripts in attachments seem to be the most throttled overall.

Regarding the efficiency of my sim ... there are a *lot* of objects that need to have their state updated every so often. It's already relatively efficient because objects do not access HTTP unless a change is actually required to the state. I'm sure it could be made even more efficient, but maybe not in a coder-friendly way. What I've been testing is more like a worst-case-scenario where all the objects need their states updated at the same time. 0.8.x always passed with flying colors. I definitely don't think Opensim should be adjusted to 'bad use cases' but in my view it should be flexible enough handle a wide variety of deployment scenarios. It served me very well before.

(0032858)
paela argus (reporter)
2018-08-13 20:44

if we evolve on the last robust date can you make sure "confirm" that the operation of the simulators in 8.1.X will not be broken?

- Issue History
Date Modified Username Field Change
2018-07-19 07:40 eform New Issue
2018-07-21 06:49 paela argus Note Added: 0032808
2018-07-21 09:02 BillBlight Note Added: 0032809
2018-07-21 09:10 BillBlight Note Added: 0032811
2018-07-21 09:12 BillBlight Note Edited: 0032811 View Revisions
2018-07-21 11:48 paela argus Note Added: 0032814
2018-07-21 11:50 BillBlight Note Added: 0032815
2018-07-21 11:52 BillBlight Note Added: 0032816
2018-07-21 11:57 BillBlight Note Added: 0032817
2018-07-21 12:05 BillBlight Note Added: 0032818
2018-07-21 12:08 BillBlight Note Edited: 0032818 View Revisions
2018-07-23 06:41 eform Note Added: 0032823
2018-07-23 06:43 eform Note Edited: 0032823 View Revisions
2018-07-23 06:43 eform Note Edited: 0032823 View Revisions
2018-07-23 07:01 BillBlight Note Added: 0032825
2018-07-23 07:18 tampa Note Added: 0032827
2018-07-24 01:04 eform Note Added: 0032830
2018-07-24 01:05 eform Note Edited: 0032830 View Revisions
2018-07-24 01:05 eform Note Edited: 0032830 View Revisions
2018-08-13 20:44 paela argus Note Added: 0032858


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker