Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0006412opensim[REGION] OpenSim Corepublic2012-11-08 19:222014-07-29 13:42
Reporterjustincc 
Assigned Tojustincc 
PriorityhighSeveritymajorReproducibilityN/A
StatusclosedResolutionfixed 
PlatformOSOS Version
Product Versionmaster (dev code) 
Target Versionmaster (dev code)Fixed in Version 
Summary0006412: Slow response for some viewers after upgrade from OpenSimulator 0.7.3 to 0.7.4
DescriptionThis mantis exists for examination of an issue reported from various sources of a significant degradation in performance for some viewers after an upgrade from OpenSimulator 0.7.3 to 0.7.4. I have not encountered this problem myself but I am interested in getting to the bottom of it.

This mantis report exists for this issue only and must not be used for other performance reporting issues. If you are suffering from it then please add your information below. I will require exact data (including steps to reproduce) and I will need you to try the actions below.

My chief suspicion is that if this doesn't turn out to be some networking issue, is that it has been triggered by enabling HTTP inventory CAPs by default in 0.7.4. Unfortunately, this is likely to be necessary since various modern viewers (e.g. LL 3.3.x) broke the fallback to UDP inventory. This may or may not be the case also in various TPVs. More information on this would be welcome.

The inventory CAP by itself is not a problem - in the long term requesting inventory data via HTTP is vastly more efficient than via the UDP connection, particularly in the case of large inventories. However, it appears that some viewers are very aggressive in fetching inventory information, particularly for active gestures [1].

It's possible that this will be alleviated by git master 61b816e, where fixes are made so that OpenSimulator's updating of inventory folder version numbers matches that expected by the client. This may lead to much more use of the cached viewer-side inventory information, reducing the request load. This may require a viewer cache clear or 2 relogs.

Thus, the first thing for anybody suffering this issue to do is to test with this commit and see if this improves the situation.

The second thing is to see if disabling the HTTP inventory capability has any effect. This can be done by adding

[ClientStack.LindenCaps]
    Cap_FetchInventoryDescendents2 = ""
    Cap_FetchInventory2 = ""

to OpenSim.ini to override the settings in OpenSimDefaults.ini. Use the "config show ClientStack.LindenCaps" region console command to check that this has been successful.

Or HTTP inventory can be disabled manually in a particular viewer via the Development menu, I believe.

Finally, if anybody does revert to 0.7.3., then confirmation that this makes the problem go away would be valuable.

[1] http://lists.berlios.de/pipermail/opensim-users/2012-November/011172.html [^]
TagsNo tags attached.
Git Revision or version number
Run ModeStandalone (1 Region) , Standalone (Multiple Regions) , Grid (1 Region per Sim) , Grid (Multiple Regions per Sim)
Physics EngineBasicPhysics
EnvironmentUnknown
Mono Version2.10
Viewer
Attached Files

- Relationships

-  Notes
(0023035)
aiaustin (developer)
2012-11-09 03:51
edited on: 2012-11-09 04:03

Openvue grid now updated to r/20937 9-Nov-2012 (git master 61b816e) for core Robust services and all add on region servers for tests.

Justin, will the second method be relevant to add-on regions for OSGrid even if their core Robust services are unchanged? I note that the standard OSGrid region distribution OpenSim.ini enables these caps set to localhost, but the comment warns this could lead to poor performance...

[ClientStack.LindenCaps]
    ; These are enabled by default to localhost. Change if you see fit.
    Cap_GetTexture = "localhost"
    Cap_GetMesh = "localhost"
    ; This is disabled by default. Change if you see fit. Note that
    ; serving this cap from the simulators may lead to poor performace.
    Cap_FetchInventoryDescendents2 = "localhost"
    Cap_FetchInventory2 = "localhost"

(0023036)
aiaustin (developer)
2012-11-09 06:46
edited on: 2012-11-09 06:51

I did a few quick tests on OSGrid add in regions with last OSGrid add on region version of OpenSim.exe (OpenSim 0.7.5 Dev OSgrid 0.7.5 (Dev) 36bfd36: 2012-10-23(Win/.NET)).

With Cap_FetchInventoryDescendents2 = "localhost"
     Cap_FetchInventory2 = "localhost"

I still immediately see many slow posts even after one avatar only has visited the region. Once locked up I get cloud avatar that persist for over an hour. E.g., response took 1,639.305 seconds? Really!

14:11:57 - [BASE HTTP SERVER]: Slow handling of 1224 POST /CAPS/d00244a6-a1b3-45
06-9d5d-ce03c6562d64 FetchInventoryDescendents2 ce614657-a331-4c2e-9ddf-789ae647
a3bc from 82.41.221.219:54204 took 1639305ms

Then I switched OpenSim.ini to include:
     Cap_FetchInventoryDescendents2 = ""
     Cap_FetchInventory2 = ""

I see some new errors related I think to an AO (not the Firestorm inbuilt AO which I don't use):

14:20:00 - [SCRIPT INSTANCE]: Unable to load script state file ScriptEngines\uuid1\uuid2.state for script ZHAO II INTERFACE uuid3 in AO...

The avatar is a cloud and that persists... so maybe setting the caps to "" just does not work with Firestorm 4.2.2?

If I teleport to Wright Plaza the avatar rezzes.. but then I cannot get back into the add on region with the caps set to "". And I still see some slow posts in there anyway (e.g. 108 seconds). These messages are on the console as I attempt to teleport back to the add-on OSGrid region...

14:33:50 - [EVENT MANAGER]: Delegate for TriggerOnIncomingSceneObject failed - c
ontinuing. Object reference not set to an instance of an object. at OpenSim.
Region.CoreModules.Framework.EntityTransfer.HGEntityTransferModule.OnIncomingSce
neObject(SceneObjectGroup so) in d:\OSGRID\OSGRID 0.7 RELEASES\osgrid.opensim-10
232012.v0.7.5-36bfd3667c44f7cd4e66e880c45c34903ed34842\OpenSim\Region\CoreModule
s\Framework\EntityTransfer\HGEntityTransferModule.cs:line 162

14:33:50 - [BASE HTTP SERVER]: Slow handling of 82 PUT /agent/uuid/ from 74.63.216.50:46256 took 108234ms

14:33:52 - [LLCLIENTVIEW]: Caught exception while processing OpenMetaverse.Packe
ts.CompleteAgentMovementPacket for Avatar Name, Object reference not set to
an instance of an object. at OpenSim.Region.Framework.Scenes.SceneObjectGroup
.CreateScriptInstances(Int32 startParam, Boolean postOnRez, String engine, Int32
 stateSource) in d:\OSGRID\OSGRID 0.7 RELEASES\osgrid.opensim-10232012.v0.7.5-36
bfd3667c44f7cd4e66e880c45c34903ed34842\OpenSim\Region\Framework\Scenes\SceneObje
ctGroup.Inventory.cs:line 70

(0023053)
justincc (administrator)
2012-11-09 19:52

OSGrid have now update the ROBUST servers to a level which includes the inventory version numbering fix so you may want to try again (with and without CAPs).

Other than that, it's much more difficult to diganose problems with OSGrid connections since there are many more factors at work (such as the current asset migration). Ideally, I need data that comes from simpler grid setups. I can still work with this but it will take longer (the next step is probably going to be for me to add some diagnostic information collection).
(0023054)
justincc (administrator)
2012-11-09 19:53

I'm not sure what's going on with the exceptions, though these look like to be attachment related. It's possible that not-providing the CAP on your grid may be causing issues with your viewer downloading appearance links (hence the cloud), leading to later problems when crossing to a new region.
(0023182)
albertlr Landar (reporter)
2012-11-29 10:01

The cap settings are definitely behind the lag and problems I have seen on my various regions. I am running my servers on a Windows 7 64 bit computer. It was so bad about two weeks ago, that I could not teleport from of my regions to another, and logging in directly was impossible. So following suggestions you had made I set mine to disabled, and the problems went away. I recently put the changes back, and things seemed to improve, but this past week the end of November it started causing delays again, but not as bad. So I have disabled them once more, again with the problems clearing up.
(0023185)
Richardus Raymaker (reporter)
2012-11-30 15:55
edited on: 2012-12-03 18:35

Ok i test this settings on 5 sims now

    ; These are enabled by default to localhost. Change if you see fit.
    Cap_GetTexture = "localhost"
    Cap_GetMesh = "localhost"
    ; This is disabled by default. Change if you see fit. Note that
    ; serving this cap from the simulators may lead to poor performace.
    Cap_FetchInventoryDescendents2 = ""
    Cap_FetchInventory2 = ""

But i have seen already a few slo requests. isupsect its maby this lines.

    Cap_GetTexture = "localhost"
    Cap_GetMesh = "localhost"

is it save to disable Cap_GetTexture and Cap_GetMesh to ?
Testing this on osgrid.

(0023197)
justincc (administrator)
2012-12-04 14:32

@Richardus - If you disable GetTexture in theory viewers will revert back to the UDP mechanism asset fetch mechanism. Whether this works correctly or not is another matter.

I suspect meshes were only ever served via HTTP so I don't know whether you'll ever see any mesh with this disabled.
(0023267)
Emperor (reporter)
2012-12-16 06:19

Okay I was having the same problem with slow post errors also and I am not connecting to osgrid. So once i fixed the problem relating to people getting stopped at region handshake thanks to justin and chris/ fleep I turned my attention to this. So I am not sure if this will help you or not but it so far seems to be negating the errors with slow messaging.

change

    ; These are enabled by default to localhost. Change if you see fit.
    Cap_GetTexture = "localhost"
    Cap_GetMesh = "localhost"

from "localhost" to your IP Address provided by your ISP. I cant explain why you would get slow post messages with localhost but it appears it does the same as 0.0.0.0 being the InternalIP in the regions ini file. The best guess I can give is localhost and 0.0.0.0 are not really a good means of binding to your ip in opensim same with SYSTEMIP for that matter.
(0023271)
nebadon (administrator)
2012-12-16 07:25

Emperor we just tested your theory and are completely unable to make it work properly in the manner you have suggested, can you please try testing this one more time for us, but this time make sure you clear your viewer cache and make sure the inventory actually downloads properly with a cleared cache, if it does work for you still, can you please paste for us exactly how you changed it so we can retest using your exact method.
(0023272)
Emperor (reporter)
2012-12-16 07:48

Sure. my team is testing that today along with a number of other tests. This also occurs for me when loading iars and oar files. I also found that increasing the max packets in mysql does stop the errors relating to mysql but i suspect this is just an issue with the viewer not being able to resolve the "localhost"

I'll let you know as soon as we complete our tests on this.
(0023273)
Emperor (reporter)
2012-12-16 12:16

Okay so per Justin's request that we retest this, we have done so. Our method does appear to work to a degree though it is not the ultimate fix. SO here is the method we used.

How we configured our region.ini file

[SG Welcome Center]
RegionUUID = f8ef2872-0a80-42a4-b5b5-b6d296bdebed
Location = 1000,1000
InternalAddress = 192.168.1.xx
InternalPort = 9001
AllowAlternatePorts = False
ExternalHostName = 67.253.72.xx

For security reasons i substituted the last digits of the two IP addresses with x. This configuration appears to work best with anyone using a FQDN as well and are behind a network router as part of a local area network.

In the OpenSim.ini file we changed the "localhost" entries to our simulator's external IP address.

[ClientStack.LindenCaps]
    ;; For the long list of capabilities, see OpenSimDefaults.ini
    ;; Here are the few ones you may want to change. Possible values
    ;; are:
    ;; "" -- empty, capability disabled
    ;; "localhost" -- capability enabled and served by the simulator
    ;; "<url>" -- capability enabled and served by some other server
    ;;
    ; These are enabled by default to localhost. Change if you see fit.
      Cap_GetTexture = "67.253.72.xx"
    Cap_GetMesh = "67.253.72.xx"
    ; This is disabled by default. Change if you see fit. Note that
    ; serving this cap from the simulators may lead to poor performace.
    Cap_WebFetchInventoryDescendents = ""

Its not the perfect fix but it does stop the console from being chatty about the slow response posts. I suspect part of it is how the UDP packets are being handled by the viewer.
(0023767)
aiaustin (developer)
2013-04-08 08:09
edited on: 2013-04-08 08:14

Just to report that I am still seeing this problem on an OSGrid add on region setup, using SQLite as the data base with the latest 30-Mar-2013 OSrid release of OpenSim.

I get many VERY long FetchInventoryDescendents2 calls and the avatar stays as a cloud once it starts to occur.. though the attachments look like they gradually load and start, but the clothing does not show even leaving it for 5 minutes or more. The viewer outfits tool show none as active, and the wearing tab under outfits says "loading" even though some attachments are clearly worn (can be seen as HUD items) and their start up messages appeared.

If I log off the avatar, the [BASE HTTP SERVER]: Slow handling continue for quite a while afterwards. Once I get this problem I can only (temporarily) resolve it with a restart of the add on regions openSim.exe. Then after the rerstrat, and reentering the viewer (Firestorm OS version) if I replace the main 4 body parts I can start afresh.

E.g.

15:59:49 - [BASE HTTP SERVER]: Slow handling of 273 POST /CAPS/45c4d7b1-7201-481
1-b721-62b1e875fc9b FetchInventoryDescendents2 ce614657-a331-4c2e-9ddf-789ae647a
3bc from 129.215.219.52:49719 took 95785ms

...


16:02:37 - [BASE HTTP SERVER]: Slow handling of 310 POST /CAPS/45c4d7b1-7201-481
1-b721-62b1e875fc9b FetchInventoryDescendents2 ce614657-a331-4c2e-9ddf-789ae647a
3bc from 129.215.219.52:49737 took 209759ms
Region (root) #

(0023837)
aiaustin (developer)
2013-05-08 09:15

Still seeing this issue. Could it be that the recommended OpenSim.ini settings to "localhost" for Cap_FetchInventoryDescendents2 and Cap_FetchInventory2 are causing the problems?

    Cap_WebFetchInventoryDescendents = ""
    ; This is disabled by default. Change if you see fit. Note that
    ; serving this cap from the simulators may lead to poor performance.
    Cap_WebFetchInventoryDescendents = ""
    Cap_FetchInventoryDescendents2 = "localhost"
    Cap_FetchInventory2 = "localhost"
(0023980)
Marcus Llewellyn (reporter)
2013-06-01 11:36

For those of us that've seen this issue occur on OSgrid, it has been suggested that older and/or slower routers could be a cause or contributor to this issue. I had a rather old Linksys WRT54G router, and recently changed to an Asus RT-N16. The Asus router is a very, very capable consumer targeted router, and this problem persists on my simulators with no perceptible change.

Just wanted to note this.
(0023981)
aiaustin (developer)
2013-06-01 12:11

The system I am reporting seeing this issue on for an OSGrid add on regions OpenSim.exe instance is on a gigabit conection to a fast switch on a university network.
(0024002)
Marcus Llewellyn (reporter)
2013-06-05 17:51
edited on: 2013-06-06 09:26

Commit e53b62-r/22826 on June 5, 2013 may solve this issue. It appears to have done so in my case.

(Edit: Got the date very wrong. Oops.)

(0024007)
aiaustin (developer)
2013-06-06 12:42
edited on: 2013-06-06 12:44

Looks like the update makes no diffeernce on my setup. I have separate MySQL and SQLite add on region setups to OS-Grid using today's r/22834 OSGrid release and everything restarted.

On both MySQL and SQLite versions, I get very many messages of this form just starting up.... no viewers comnected at all and nothing else running on the Windows Vista system.

20:30:01 - [SynchronousRestObjectRequester]: Slow request 801 GET http://assets. [^]
osgrid.org/assets/113ad401-3293-4ac1-a2ca-8f1ffdf4326e took 4462ms, 0ms writing,

20:30:05 - [SynchronousRestObjectRequester]: Slow request 802 GET http://assets. [^]
osgrid.org/assets/e7df80df-9fcc-4232-9b25-e53439804845 took 3854ms, 0ms writing,

20:30:09 - [SynchronousRestObjectRequester]: Slow request 803 GET http://assets. [^]
osgrid.org/assets/b66b6581-22cb-4ea3-9815-69821ef679f2 took 4290ms, 0ms writing,

20:30:21 - [SynchronousRestObjectRequester]: Slow request 818 GET http://assets. [^]
osgrid.org/assets/92cf1304-f69b-43f1-a77d-f5771f550464 took 4118ms, 0ms writing,

The host computer for the add on reguion instanmce of OpenSim.exe is one identical to several others running the Openvue grid region servers which don't seem to give this error.

When I connected a viewer (Firestorm 4.4.0 OS), on another separate system so as not to load the OpenSim server machine, I get message as before about Slow handling...FetchInventory2

20:38:14 - [BASE HTTP SERVER]: Slow handling of 41 POST /CAPS/77e21dc7-3045-4bfd
-9fe6-cb68fd3ccf4a FetchInventory2 e858df02-a860-4b92-937a-2b87e4ebcd6d from 92.
237.199.112:50147 took 5007ms

(0024015)
Marcus Llewellyn (reporter)
2013-06-06 14:59

I do see just a few slow requests from assets.osgrid.org when I start a simulator. But they're very few, and disappear once the cache gets a grip on things.

Since Melanie's inventory throttling commit, I've seen no slow requests for the FetchInventory2 cap. This is a definite change from before the commit, when I would get a torrent of these. With more than one agent trying to load inventory from my simulators, I would even experience a slow-down so severe that avatars could not walk around, and requests that reached hundreds of thousands of milliseconds in duration. Again, inventory throttling appears to completely relieve that issue in my case.

I have tested with Kokua, Singularity, and Firefox. My simulators, MySQL server, and viewers are each on separate machines. None of these viewers produced slow handling messages for FetchInventory2. Kokua and Singularity loaded inventory in a quick and snappy fashion. Firestorm did not load inventory with nearly the same celerity.

- Issue History
Date Modified Username Field Change
2012-11-08 19:22 justincc New Issue
2012-11-08 19:22 justincc Status new => assigned
2012-11-08 19:22 justincc Assigned To => justincc
2012-11-08 19:38 justincc Summary Slow network response for some viewers after upgrade from OpenSimulator 0.7.3 to 0.7.4 => Slow response for some viewers after upgrade from OpenSimulator 0.7.3 to 0.7.4
2012-11-09 03:51 aiaustin Note Added: 0023035
2012-11-09 03:58 aiaustin Note Edited: 0023035 View Revisions
2012-11-09 04:03 aiaustin Note Edited: 0023035 View Revisions
2012-11-09 04:03 aiaustin Note Edited: 0023035 View Revisions
2012-11-09 04:03 aiaustin Note Edited: 0023035 View Revisions
2012-11-09 06:46 aiaustin Note Added: 0023036
2012-11-09 06:47 aiaustin Note Edited: 0023036 View Revisions
2012-11-09 06:48 aiaustin Note Edited: 0023036 View Revisions
2012-11-09 06:49 aiaustin Note Edited: 0023036 View Revisions
2012-11-09 06:50 aiaustin Note Edited: 0023036 View Revisions
2012-11-09 06:51 aiaustin Note Edited: 0023036 View Revisions
2012-11-09 19:52 justincc Note Added: 0023053
2012-11-09 19:53 justincc Note Added: 0023054
2012-11-29 10:01 albertlr Landar Note Added: 0023182
2012-11-30 15:55 Richardus Raymaker Note Added: 0023185
2012-11-30 15:56 Richardus Raymaker Note Edited: 0023185 View Revisions
2012-11-30 15:57 Richardus Raymaker Note Edited: 0023185 View Revisions
2012-12-03 18:35 Richardus Raymaker Note Edited: 0023185 View Revisions
2012-12-04 14:32 justincc Note Added: 0023197
2012-12-16 06:19 Emperor Note Added: 0023267
2012-12-16 07:25 nebadon Note Added: 0023271
2012-12-16 07:48 Emperor Note Added: 0023272
2012-12-16 12:16 Emperor Note Added: 0023273
2013-04-08 08:09 aiaustin Note Added: 0023767
2013-04-08 08:10 aiaustin Note Edited: 0023767 View Revisions
2013-04-08 08:14 aiaustin Note Edited: 0023767 View Revisions
2013-05-08 09:15 aiaustin Note Added: 0023837
2013-06-01 11:36 Marcus Llewellyn Note Added: 0023980
2013-06-01 12:11 aiaustin Note Added: 0023981
2013-06-05 17:51 Marcus Llewellyn Note Added: 0024002
2013-06-06 01:24 aiaustin Note Added: 0024003
2013-06-06 09:26 Marcus Llewellyn Note Edited: 0024002 View Revisions
2013-06-06 12:42 aiaustin Note Added: 0024007
2013-06-06 12:44 aiaustin Note Edited: 0024007 View Revisions
2013-06-06 12:45 aiaustin Note Deleted: 0024003
2013-06-06 14:59 Marcus Llewellyn Note Added: 0024015
2013-12-05 18:10 justincc Status assigned => resolved
2013-12-05 18:10 justincc Resolution open => fixed
2014-07-29 13:42 chi11ken Status resolved => closed


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker