Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008396opensim[GRID] Asset Servicepublic2018-10-17 10:562018-11-07 10:24
ReporterSnootsDwagon 
Assigned ToUbitUmarov 
PriorityurgentSeveritymajorReproducibilityalways
StatusclosedResolutionwon't fix 
PlatformOSOS Version
Product Version0.9.0.1 
Target VersionFixed in Version 
Summary0008396: Texture priority not set properly
DescriptionWhen rezzing a box it is understandable that object is the current focus of user attention. It would be appropriate then that the textures on that box should immediately take top priority in rezzing-- and should do so almost instantly. Instead it can take up to 15, 20 seconds or more for the texture on a box to rez.

The same holds true for a sign that is zoomed in on or right clicked. Since that is the obvious focus of the user, it is sensible that such an item would be the top rezzing priority at the moment, overriding all other texture loading (basically, butting to the head of the line). Currently that is not the case, and users find themselves waiting for significant periods of time to read a sign, or to see a texture on a box.

"If the 2D don't work... the 3D don't work." ; )
Steps To ReproduceRez any box from inventory on a standard region. I am currently using the latest OSgrid Opensim installation. The texture will on very rare occasions rez almost immediately (which is as it should do). But 99% of the time it will take ages to rez. On occasion it fails to rez (leaving a gray box) or fails to rez fully (leaving a discernible picture but "fuzzy").
Additional InformationThis is not just a texture priority issue... there is a more general texture loading issue where some textures fail to load or take a very long time to load, whether viewer focus is on them or not.
TagsNo tags attached.
Git Revision or version number
Run Mode Standalone (Multiple Regions)
Physics EngineBulletSim
EnvironmentMono / Windows
Mono VersionNone
ViewerFirestorm
Attached Files

- Relationships
related to 0008404new ON-DEMAND priorities not set for best user experience 

-  Notes
(0033172)
Sheera Khan (reporter)
2018-10-18 01:18

Hi Snoots,
as much as I agree with that - but wouldn't that be a viewer side issue? I'd guess the viewer sends the request for the textures to the server and the server will answer them more or less in the order of requests (give or take a little for multiple parallel streams). Maybe the viewer puts too much into the request queue so the now most relevant texture to load has to wait until the server even gets to see the request.

But as I said, it is just a guess of mine. I didn't look into the viewer side code. At the developer meeting (every tuesday at 19:00UTC) you can meet developers with insight into server- and client-side code.
(0033174)
Luisillo_Contepomi (reporter)
2018-10-18 09:44

I can not reproduce the problem Opensim 0.9.0.1 and Firestorm 5.1.7.55786
Network cache Bandwidth configured (as always for me) 3000Kbps and 9984Mb disk cache.
All texture is shown immediately on some face of a new object.
(0033175)
SnootsDwagon (reporter)
2018-10-18 10:15
edited on: 2018-10-18 11:10

Hi Sheera, Luisillo. Sheera you may be right. This is the old "is it the viewer or the OS" question... and as a user I don't know. I just report the bug. It affects OPENSIM and all related grids, so whether it's opensim or viewer code, it's still of immense import to the Opensim project. Failed textures is pretty much one of the top three issues reported on the Mantis over the years. I'm reporting it still exists after the recent major update.

Luisillo... no offense intended-- I find it VERY hard to believe you cannot reproduce this problem, as it is a problem noticed by almost every user of virtual worlds since the beginning of Second Life. This has been tracked, documented, data-proved, and a huge debate with Linden lab themselves way back in 2007 (they finally "discovered" a severe bottleneck in 2010). The problem still exists (I am a retired computer analyst and have been tracking this issue-- along system techs such as Balpien Hammerer-- since 2005).

So the question would be: why are you NOT experiencing this issue? Is it your computer configuration? Some special setting on your system that no one else has done? In that regard:

Are you using a high-power gamer-level computer with super-graphics card or a standard, every-day type computer?

What is the graphics level setting of your viewer? Any special settings?

Is there anything on your system to which you could attribute uncommonly over-the-top performance?

That all textures always rez for you puts you somewhat in an extremely rare class of people. The vast majority of users experience these issues. If you don't-- then additional information as to WHY that might be the case could prove very helpful in the discovery process. : )

You claim you "can't replicate" this issue. So the primary question is-- why can't you? If you truly can't replicate this issue... the whole Opensim system needs to find out the core reason so we can duplicate that.

Just to make sure: Are you saying that every time you call up a photo from your inventory it almost immediately appears, no lag time, never fails to fully rez?

Are you stating that when you apply a texture to a prim it applies immediately, without any delay, fully rezzed? If so, are you using a texture sorter? (which displays textures prior to actual use.)

Are you stating that when you enter a region... all textures rez, no gray items, no fuzzy textures, ever, even when zooming in close?

Just wanting to make sure on all these things, that we're on the same page of what this Mantis is reporting. Thanks for any information you can provide.

(0033180)
Luisillo_Contepomi (reporter)
2018-10-19 07:04
edited on: 2018-10-19 07:06

For tests I am using as server my old computer (from 2012 )

CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (3395.65 MHz)
Memoria: 16345 MB
Versión del Sistema Operativo: Microsoft Windows 7 SP1 64-bit (Build 7601)
Fabricante de la tarjeta gráfica: NVIDIA Corporation
Tarjeta gráfica: GeForce GT 640/PCIe/SSE2

Versión de Windows Graphics Driver: 24.21.14.1163
Versión de OpenGL: 4.6.0 NVIDIA 411.63

I am running mysql 8.0.12 server Robust, and a region in grid in the same computer.

I do the connection with a last version Firestorm in other old computer (from 2011) connected in a 100Mb local network. Network cache Bandwidth configured (as always for me) 3000Kbps and 9984Mb disk cache.

I am a tester and I can not reproduce the problem as you as you said:

"Rez any box from inventory on a standard region. I am currently using the latest OSgrid Opensim installation. The texture will on very rare occasions rez almost immediately (which is as it should do). But 99% of the time it will take ages to rez. On occasion it fails to rez (leaving a gray box) or fails to rez fully (leaving a discernible picture but "fuzzy"). "

I am sorry but I can not reproduce this as you said. I'm lucky.
(I can answer all your questions if you are in the users mail list. This is a mantis.)

(0033181)
SnootsDwagon (reporter)
2018-10-19 08:40
edited on: 2018-10-19 08:45

Luisillo (if I understand your description correctly), perhaps rather than testing on your own Robust server (which has its own separate asset server system) you should test on a public test grid such as OSgrid. Using an abnormally fast I7 system with 16 gigs RAM on a local mini-grid server isn't going to produce standard user results.

I assure you the average user of a public grid can easily reproduce this issue. To be of value to the Mantis, one has to test these things under normal conditions, not an enclosed limited-access environment.

(0033196)
Garmin Kawaguichi (reporter)
2018-10-19 12:57

The first attention to have for a lag problem generally and gray textures in particular is the link between the viewer and the server:
- avoid using Wi-Fi
- avoid using a VPN
- avoid using a low-end proxy

Then, even if you have a viewer connected at very high speed and at the other end a very high speed server, the transfer speed (as well as the loss of request packets) is only affected by the weakest hardware constituting the link chain.

Use flow testing and transfer quality testing equipment between OSGrid.org and the viewer.

T LB
(0033268)
SnootsDwagon (reporter)
2018-10-26 11:10
edited on: 2018-10-26 11:20

"Use flow testing and transfer quality testing equipment between OSGrid.org and the viewer."

Is this kind of equipment-- and the knowledge of how to use it-- available to the casual user?

The problem still exists of textures taking far too long to load, or not loading at all. I have a slideshow viewer that is designed to eliminate texture lag. I wrote the script myself, yet even that scripted-to-be-zero-lag device has trouble getting textures to rez.

Whatever is causing this problem, it affects Opensim installations throughout the metaverse, everywhere. With all repect, this is one of those things where I recommend people stop passing the buck and get down to fixing what's causing this problem. It's one of the top-most issues of using virtual worlds, and has been ever since the founding of Second Life.

As a user, it is understandable I cannot ascertain whether this problem is caused by Opensim, OSgrid, Kitely, 3rd Rock, DigiWorldz some other grid, this viewer, that viewer, Robust or localhost. I'm interested in my virtual region and devices operating correctly. So I have to leave it in the hands of the devs-- of whatever software-- to figure out where the bottleneck is and remove that bottleneck.

Because honestly... textures not rezzing properly has gone on far past long enough. And I'm finding it difficult to believe it is 100% a viewer problem with zero issues server-side. Testing this on a sealed localhost server is not proper "ground zero" testing (which proper debugging procedure demands). So difficult to tell where the bottleneck is exactly, but I do believe someone needs to figure out where this problem is and eliminate this issue completely. Way past time this is fixed.

(0033273)
aiaustin (developer)
2018-10-27 01:05
edited on: 2018-10-27 01:21

I had one avatar on one grid (latest dev master) on one region after settling down for an hour and with no other adtivity and after login sat looking at one texture, pretty much the only thing visible in the viewer (Firestorm) , and it went to its blurry state after 10 seconds or so and then I sat watching it with no activity on the servers, data base or viewer I could see for over 3 minutes before it crisped up. It did not use to be as bad as this as I recall.

Also on a wall of textures in a store style setup, I see lots of the textures go blurry and then take a long time to crisp up. Could it be that far too many partial downloads start and then somehow get jammed until somthing forces them to retry. They eventually do crisp up if you leave things for a very long time.

No VPN, all on same local network, very fast gigabit ethernet wired links. I cannot think its the network.I have had local sysadmins check for network and port issues and they can find nothing. I cannot though exclude a really problematic Windows issue, as things do seem to have got a lot worse since about April 2018 which was a time of change for OpenSim to 0.9.1 dev master, .net4.6 and Windows 10 update to winver 1803.

I am investigating (and reporting on in other Mantis issues) very seious content loading lag for prims and mesh also, and a lot of TP failures. even on a trivial single grid with two regions in my testing. I do suspect these are linked as there is so little activity on my test networks when the failures and slow downloads are happening. I am hoping one day we will hqae an “ah ah” moment and find some sort of timing issue.. just hoping. Its all a bit of a mess at the moment and has been since April 2018.

So I encourage anyone having these and related issues to begin noting them on the relevant Mantis entries and Viewer JIRA/issue trackers to try to get a handle on when it does and does not occur. And ask devs and other reporters to be open minded and see the reports as indicative of some issue or other wherever er they appear.

(0033275)
SnootsDwagon (reporter)
2018-10-27 06:43

Those things you mention aiaustin are very interesting, as they have a visible commonality: failure to verify completed transfer.

You mentioned TPs. I had the unusual situation of being able to watch two different servers (one local, one using TeamViewer) during a teleport. The origin server sent the avatar information, the second server stated the full data wasn't received-- and then the first server dropped the avatar as if the action were completed. Visible cause: improper handshake routines. As you state, that isn't Net, that isn't viewer: that is direct handshake bugs in the Opensim code. Interestingly those issues vanished (to my experience) with the latest release of the OSgrid dev code. However, the texture issues remained.

I believe similar problems are happening on textures. They are initialized, then the system simply loses track of the fact they have not rezzed properly. In addition, priority is not being given to the textures in front of the avatar, and for a certainty texture cache isn't being handled correctly.

Just the other day in a test I had two texture panels: one in front of me and one behind me. I would look at one and after far too long a time the texture would finally rez. I would then turn my back to it to look at the other, and the same long rezzing period would be experienced. THEN on turning back to the first texture-- which should have either already been there or loaded *immediately* from cache... the texture was greyed out and took just as long to load all over again. This process repeated as I turned back and forth from texture to texture. Two textures. There is definitely a severe texture tracking and loading logic bug here.

This is the very same problem Balpien Hammerer and I discovered way back in 2005 and 2007 on Second Life-- redundant texture loading that causes excess strain on the asset server. My experiment was just one instance. Multiply that by 1000 avatars several times a second and we create a texture loading cascade, creating the equivalent of an internally-created DDoS attack on our own servers. Its no wonder the system bogs down from time to time.

The problem: we have Opensim code, Robust code and Viewer code-- and which of these is responsible for having a hand in this issue? That's the $20 question. This question is exacerbated by the tendency of people to pass the buck onto other dev teams. "Oh that's a viewer problem" is something I hear far too often. Because as a user *I don't care* where the problem originates. As users... we need it FIXED, no matter where it originates. The fact that Firestorm has disavowed Opensim was a matter discussed at the latest Opensim dev meeting-- the great need for Opensim to get some Viewer volunteers to take over viewer development specifically for Opensim so that we can get away from the Firestorm non-cooperative bottleneck.

The bottom line is these problems definitely exist. You perfectly described everyday user experience in your post aiaustin. The question: who is going to stand in front of a sign or a vendor for 3 minutes plus waiting for it to rez? Who wants to right click on every. single. sign. they see in order to make it take focus and rez faster? What kind of new user experience is this going to present?

This is why Mantis reports like this are needed: they not only point out the problem-- we point out the impact on user experience that makes more vivid WHY the bug is so serious. Failed texture rezzing affects every aspect of user experience-- experienced and beginner-- every day. "If the 2D don't work, the 3D don't work".

When we realize that sculpties are texture based, we realize how far this impacts the system. A slow-to-rez texture means a non-rezzed (invisible) sculpty. If textures rezzed quickly, sculpties would be very valuable tools.

Thus, this Mantis... and the continued *stressing* of how serious this problem really is. It affects every aspect of virtual world use. If we can't SEE the virtual world... how can we use it? Or do we enjoy walking around among greyed out building, un-rezzed, unreadable signs, and vendors that can't be used?

Testing these things on closed-box, localUser systems is not effective debugging method. It eliminates all the problems one has to deal with in "real life road conditions". It limits the environment to one's specific instance rather than testing the problem on the front lines, at ground zero... using the kinds of computer the everyday user will have in their homes (which for the most part, is not an octacore I7 with 32 gigs of memory and a $500 graphics card). Which is why *debugging method* is called into question here as well.

If we're going to debug code... we have to venture out to where the bugs are. ;)
(0033276)
aiaustin (developer)
2018-10-27 07:19
edited on: 2018-10-27 07:24

All too familiar... but remember the issues began for me on dev master based grids in March/April 2018 and were not present on my regularily updated grids before then.

I can add that the TP failures are still present in the very latest dev master code in my testing even on a very simple grid, no HG travel, etc as reported elsewhere.

(0033277)
SnootsDwagon (reporter)
2018-10-27 10:36
edited on: 2018-10-27 13:13

Yes, I've heard some issues were not present until the latest update. A lot of other problems vanished, a lot of things got faster. But some things totally broke-- such as the localHost function (which now requires specially updated files to install peroperly).

My TP experiencies before 9.01 were that I'd crash more than 50% of the time just teleporting from one region to another on OSgrid. It got so bad people were crashing teleporting to LBSA. That particular issue was fixed in the latest OSgrid release; I pretty much never crash on teleporting any more. That doesn't mean it's *totally* fixed... just much better than it was.

Although it's not the issue of this particular Mantis, I have a feeling these issues are related. I believe the TP problems are simple lost data / handoff failures. The rule of handoff: retain existing data and do not drop the feed until the target location absolutely verifies all a data has been received and the handoff successful. That evidently wasn't / isn't happening. Probably still needs some tweeks here and there, but for me it works a lot better than it did.

Texture loading issues have existed from the early, early days of SL, so it's not surprising they still exist and yes-- are very likely largely due to viewer issues (which is why Opensim needs to start working on its own viewer). Start with the Firestorm open source code, completely re-design the user interface, fix the major problems-- such as texture loading and cache tracking.

It's a big job to be sure and not easy. But one thing I know for a certainty: we'll never fix it by saying "It's a viewer issue"... and leaving it in the hands of a viewer dev team that has no concerns about Opensim. In such case that becomes an excuse. If it is indeed a viewer issue, then the Opensim community needs to start with the current viewer foundation and take over development specifically for Opensim. Otherwise, if we don't have a valid, functioning viewer, Opensim never will work correctly. Can't have one without the other.

(0033278)
FreakyTech (reporter)
2018-10-27 11:00
edited on: 2018-10-27 11:01

Honestly, I did a lot of examinization over the years on all things related the UDP harness. Several bugs that can be seen are based on a misunderstanding of how things should be done. Quite commonly actually being found in OpenSim's UDP handling code. The real problem with it fixing those bugs means breaking nearly all workarounds that sit on top.

The throttling code on GetTexture tends to make viewers go for that UDP harness and then texture download takes ages due to protocol understanding differences.

[OT for this Bug] The TP crashes are commonly created by a race condition that is also partly triggered by OpenSim. Mostly an ordering problem. Actually fixed to some extent on arriba fork. However, that fix is not going to apply easily to 0.9.[/OT]

(0033381)
aiaustin (developer)
2018-11-02 04:38

Please test this again with 0.9.1 dev master releases 512 (2018-11-02 02:25) and later as a fix for serious TP issues related to HTTP event handling may have a bearing on this issue.
(0033403)
SnootsDwagon (reporter)
2018-11-05 12:36
edited on: 2018-11-05 12:40

I tested this on ElvenSong directly following the rebooting of OSgrid today.

* Rezzed a simple cube
* Applied a texture from my inventory to one side at a time, total 5 sides, 5 different textures.

Findings: textures took an average of 16 seconds to load to full resolution. This was while zoomed in so that the box was the only thing in my view.

Expectation: a texture in my inventory applied directly to a box should rez very quickly.

TEST 2:
* Double clicked a texture in my inventory to view it on the screen.
* Results were highly variable, depending on texture size.
* Textures took between 3 seconds and 10 seconds to fully rez on my screen.
* Rez time did relate to texture size (resolution) but was not consistent.

Expectation: Direct from inventory to my viewer screen should be fastest texture rezzing of all. Still took up to 10 seconds to rez a texture.

In the above tests I have noticed improvement over say, 3 months ago... when textures would remain gray or fail to complete to full rez. So that's the positive side. Negative side: it still takes an average of 16 seconds to see a texture rez on a prim-- which will negatively affect photo slideshows, vendors, etc.

We can expect this experiment to vary widely, depending on the region, internet speed and computer one owns. However my region is fast (average FPS 95), I'm using a fast i5 computer (3.8ghz quad core) with 8 gigs of RAM, a GeForce 1050 card with 2 gigs dedicated RAM, and I have cable internet with a low ping rate. So if anything my rezzing speed would be predictably faster than the average user. Not as fast as ultra-gamer-level systems, but pretty fast. : )

(0033404)
BillBlight (developer)
2018-11-05 13:37

Not disputing what you are seeing but you realize that nothing comes "directly" from your inventory ..

You apply texture from your inventory, viewer makes request to the asset server on osGrid, it locates the texture sends the info back to the viewer, the viewer then applies the texture to the prim, and updates the local cache.

There are many factors involved, HD cache speed, net speed, asset server load.

So as far as "from your inventory" being faster , that is really a moot point.

Also test this on other regions, such as the test sim I have up Dev Outreach on osGrid, which is based on the the absolute newest code.
(0033405)
SnootsDwagon (reporter)
2018-11-05 14:11
edited on: 2018-11-05 14:36

Yes Bill, I'm aware of this, as are likely most people here... but it's still good you brought this up for anyone who might not understand how behind-the-scenes works.

However, the point made in "directly from my inventory" could be rephrased as: a direct instruction to the system to provide me with a specific texture.

In other words I'm not walking through a park and waiting for items to rez seemingly at random. I'm in my inventory, manually triggering a specific texture to rez. By logic that should take top priority over all other texture rezzing processes and occur "right now"... not cued in line with hundreds of other textures vying for prominence.

So yes, calling a texture "from your inventory" should be "faster"... in that it should take spot 1 in the rezzing cue list and all other rezzing processes take a back burner until that specifically-demanded texture is fully rezzed.

In addition, when all other textures are rezzed (ie, the texture-loading data window is showing no texture loading)... then directly calling a specific texture from inventory should have immediate priority. One would not expect it to take an average of 16 seconds to apply to a prim, or up to 10 seconds to appear on the screen. It's a single texture. That should not require much in the way of system resources. This is why I've chosen to label this as a bottleneck situation. Some process somewhere is either getting in the way or failing to allow the system to do what it has been directly and specifically commanded to do, namely: rez a single texture, now.

As far as testing on Dev Outreach-- I've conducted tests to my satisfaction on a server I host and can watch the result on. No need to conduct repetitious tests on a server I do not know. I was asked to repeat the tests, did so, the results are above. : )

(0033406)
BillBlight (developer)
2018-11-05 14:23
edited on: 2018-11-05 14:29

So you are saying if other people are pulling things from the asset server it should stop, and move your texture request to the top of the list?

That is not how it works ..

There is the main asset server with everyone on osGrid using it , rezzing textures, rezzing items, telporting, making notecards, sending IM's .


I tested this on my test region on osGrid and a texture from my inventory appears on the prim in under 5 seconds ...


And just to note when I say I can't replicate something, that means my keyboard is not broken and I actually tried and tested it myself.

(0033407)
SnootsDwagon (reporter)
2018-11-05 14:32
edited on: 2018-11-05 14:37

The bug is reported, tests conducted, results reported. Done here.

(0033408)
BillBlight (developer)
2018-11-05 14:43

Hmmm ..

So far it appears that you have refused to test anywhere but one sim.

You have ignored the fact that others are unable to replicate this issue.

You have refused to test anywhere but osGrid (which is a development grid not a commercial one).

If you would have tested on the sim I asked you to at least then I would have logs to look at fully and a network to examine.

That is not very analytical.
So no can't help you, and done.
(0033409)
SnootsDwagon (reporter)
2018-11-05 14:59
edited on: 2018-11-06 05:49

Bill, I didn't notice until just now that you were a dev (microscopic letters and old eyes). To explain: I'd had my fill of arguing with Linden Lab over the same obvious technical issue back in 2005 and had no desire to repeat that silly process. Linden Lab found out 5 years later they were wrong and could have fixed the problem 5 years earlier if they had realized some of their customers are professionals in real life.

You say "others have been unable to replicate this issue". That is clearly not true-- as is evident by other Mantis reports of this issue. That calls for further research on your part... based on the evidence of multiple Mantis postings. I'm not the dev; you are.

If you'd said you wanted to track logs and network activity, I'd have been right there. You didn't tell me that's what you wanted to do. I had already run tests on my own stable in-house server, where I could watch the results right on the server screen, and felt no need to repeat those tests. Only so many hours in the day and I wasn't aware of what you really wanted to do. I am as analytical as they come... but I have to know what's on your mind if you want me to assist.

(0033410)
UbitUmarov (administrator)
2018-11-05 16:09

Textures will always take some time to show up.
- The fast situation is when it was seen before and is fully on the viewer cache.
- Second is if it is on the region cache.
- Worse is when it must be requested to grid assets server.
- just getting a object from inventory, most likely is 3rd case.

getting a texture is also not one region http request, it is usually 2. one to get the asset description and worse LOD (fuzzy look), second to get the needed LOD, if you move closer another one may happen to get a better LOD (similar with meshes)

And this is after the viewer decision to ask for it. Requests prioritization is done by the viewer and textures http requests may wait for other more important ones at viewer level also. Textures and meshes are not the higher priority things.

Regions and grid must treat all texture requests as equal. They can (must) also prioritize other more important requests. (a grey texture will not make you log out, other missing or delayed information may)

it is irritating to be in front of a object and it never getting high rez texture or mesh while we see others around getting them, but thats mostly viewer thing, and not new, see it a lot you konw where.
Or turning around and all grey, or avatars missing all meshes, and stuck like that for ages..
Again viewer side most the time..
(0033411)
SnootsDwagon (reporter)
2018-11-05 17:16
edited on: 2018-11-05 19:19

Agreed Ubit, on all counts. Which, if we let it rest there, this problem with textures will continue, remaining unfixed.


The number of Mantis reports as well as comments from users every day should leave no doubt there are texture problems... and that this system can be improved. No one needs me for that process. I'm a user, not an Opensim dev. I can tell the Emperor he has no clothes, but I can't force him to get dressed. ; )

(0033412)
SnootsDwagon (reporter)
2018-11-05 17:22
edited on: 2018-11-06 05:56

(As a note, I did have a great huge long technically-oriented message detailing the numerous possible issues involved in this... but the system borked and lost about an hour's worth of analysis and I realized: what am I doing? I'm supposed to be retired from this kind of stuff.) mwahahahahaa...

Bottom line is this: under no circumstance should it take an *average* of 16 seconds to rez a single texture from inventory. That it does so (regardless of claims otherwise) should have people hopping all over this. If we're serious about improving Opensim: fix the foundation issues first. You don't get much more foundation than rezzing 2D textures. There are definitely bottlenecks. If one believes this problem doesn't exist or the system can't be improved... that would be a problem of itself. Ignoring the problem won't fix it. That's why I so strongly urge this be looked into with a code magnifying glass, one end to the other. You fix this, you fix one of the major "lag" issues that have plagued Opensim for years (and indeed, Second Life itself).

(0033414)
aiaustin (developer)
2018-11-06 06:40
edited on: 2018-11-06 07:31

As others have indicated... even taking an item from inventory and trying to view it in a viewer may have a complex pathway and I do see grey image preview panes when I try to do the same on pretty much any Second Life and/or OpenSim setup occasionally.

But I thought I would visit OSGrid "ElvenSong" region and see what I can spot in the Firestorm viewer console log (ctrl+shift+4) or the Texture Debug console (ctrl+shift+3)…

ElvenSong is a 1,280m X 1,280m varregion (equivalent to 25 normal regions) and appears to have content totalling 40,183 prims/land impact objects. A SciFi region up at 3,500m sits above the ground level content. The region is heavily textured with many interesting Elven and SciFi themed areas.

First thing I spotted is that we have a LOT of this sequence of warnings... involving an image that appears to want to be 320x320 (how did that get in?) and they seem never ending even after being on the region without moving the avatar or camera for over 15 minutes and when all images in sight have crisped up...

2018-11-06T14:24:28Z WARNING: llrender/llimagegl.cpp(505) : LLImageGL::setSize: Texture has non power of two dimension: 320x320
2018-11-06T14:24:28Z WARNING: llrender/llimagegl.cpp(1314) : LLImageGL::createGLTexture: Trying to create a texture with incorrect dimensions!
2018-11-06T14:24:30Z WARNING: llmessage/lltransfermanager.cpp(309) : LLTransferManager::processTransferInfo: 3cc3b784-853a-017a-c7c4-61d21bb04a05: Non-ok status, cleaning up
2018-11-06T14:24:30Z WARNING: llmessage/lltransfertargetvfile.cpp(197) : LLTransferTargetVFile::completionCallback: Aborting vfile transfer for 4613d61e-6651-46b8-9692-a188e00f71a1

Secondly there was a LOT of activity when looking at the texture debug panel... and it seemed to go on a long time. Some of the images loading for a while were 1024x1024. I upped the cache size to 9984MB (max, nearly 10GB) from my normal 2GB and it did eventually stop with idle on the texture console. But even in a relatively static state the 320x320 wrong dimension image messages continued in the debug console. Those vfile transfer abort messages seem to repeat for just a few UUIDs...

4613d61e-6651-46b8-9692-a188e00f71a1
7be5bef4-6f85-4b2a-8ea9-61905750651f
8907d9d2-6c75-4296-9f33-1e4436193832
a707a9b7-ed3f-4fef-a5c3-cb8950681044
e5fe3eb3-4c7e-4e71-915b-6a262f1cc05a

I wonder if this triggers any thoughts to anyone?

(0033423)
SnootsDwagon (reporter)
2018-11-06 15:30

Those are very interesting findings. I too question how that 320x320 got in there. I have been seeing on the server repeated yellow-line texture warnings but didn't have a chance to check deeper into it (been pretty busy lately). I thought it was just more standard texture reloading or perhaps caused by a bug in the recent updates. I would be curious where that texture is located / triggered and how to remove it from the system.

I have noticed before (and reported on this Mantis) redundant texture loading. In one Mantis I reported standing in a closed room with my draw distance set on 32m and for 30 minutes the texture load window showed the same textures loading over and over.

Will note that I have run the "texture loading test" on sandbox regions with the same results. A friend and I once stood together on a sandbox and tried loading textures from our separate inventories-- both noting the same excessively long rezzing time.

So yup, something is off. Seems like some strange issues on ElvenSong (even though such doesn't seem to cause avatar or FPS lag). But those issues don't appear to be unique to Elvensong... just the particular instances. The issues themselves seem rather widespread, as they have been experienced elsewhere.
(0033434)
SnootsDwagon (reporter)
2018-11-06 20:30
edited on: 2018-11-07 08:23

Update:

At the urging of Bill and after reading Aiaustin's post above I went to Sandbox Plaza II today to run the same texture rezzing tests. Results:
* Applying a texture to a prim average 3 seconds to full rez.
* Viewing a texture from inventory to screen: widely varied, ranging from 2 seconds to 7 seconds. Rezzing time did not seem related to size of texture.

I was watching the ElvenSong server while performing these tests, and as Aiaustin reports above, there is all kinds of yellow and redlining that wasn't appearing before the latest update. So am curious what might be going on there. I hadn't felt it was a region-related problem because I notice no lag while using the region (other than texture rezzing). But something at least *on that region* is causing some texture bottlenecks and I'm curious what it is. The server log screen does not provide sufficient location information to track down the offending objects.

So... Bill is right, standard texture rezzing is pretty fast (at least it was on Sandbox Plaza II). Three seconds is acceptable speed. But then again, there's pretty much *nothing* on Sandbox Plaza to interfere with texture rezzing. So it it an issue of "an empty sim is a happy sim" or is it actual problems with ElvenSong?

Would appreciate any assistance that could be offered there. (OSgrid / ElvenSong region). There are definitely issues, just not sure why they have only started appearing recently. The system seems to be trying to locate information that is missing... repeatedly. That wasn't the case before the update (at least not to this extent).

My prior findings were based on measurements made on a region that appeared to be working perfectly fine... but the background information is now showing some very odd issues. Going to run tests on other random regions to see what results occur around the grid. Will report those in detail when completed.

Edit: Since there is no discernible lag and FPS is high on ElvenSong, I surmise the problem isn't with performance but a texture bottleneck, as previously considered. Further testing on other regions is warranted.

(0033435)
SnootsDwagon (reporter)
2018-11-06 21:21
edited on: 2018-11-07 08:21

I conducted a series of tests tonight on three random, populated regions. In each case I was the only avatar on the region.

Results of tests: applying a fresh texture to a prim took an average of 13 to 16 seconds in all cases. This is excessively long, especially when using an item such as a vendor or photo viewer.

Educated observation: The Opensim system is giving insufficient priority to on-demand specific texture rezzing.

On demand texture rezzing takes several forms:

* Double clicking a texture in inventory to view on the screen
* Dragging a texture from inventory to apply to an object
* Clicking a scripted item such as vendor or photo viewer
* Zooming in on a sign to better see/read the applied texture

Since these things are all the sole, 100% focus of the user at the time, prioritizing the rezzing of that texture should be very high. Opensim is not giving enough priority to on-demand texture rezzing.

This is the initial bottleneck of the system. In short, Opensim is prioritizing things that are not of immediate interest to the users. This is regardless of whether there is one or twenty avatars on the region.

Reality of virtual reality: when a user invokes an on-demand action... that action needs to be prioritized to maintain and promote user experience. Currently that is not happening. Thus users can't quickly read signs, use vendors, see photos, etc.

Evidence indicates this isn't a problem solely with ElvenSong region; it's a wide-spread issue involving Opensim event prioritization. As a core issue, Opensim simply isn't giving enough priority to direct and immediate user needs, thus slowing the user experience and seemingly creating "lag" where there is no actual lag (at least, not that has been yet measured); it's a failure to high-prioritize on-demand events.

The results of these tests is interesting in the light of the fact that the title of the original post in this mantis is: TEXTURE PRIORITY NOT SET PROPERLY.

These tests bear out this is indeed exactly what is happening. But to be more accurate: on-demand user activity not assigned due priority.

(0033436)
UbitUmarov (administrator)
2018-11-06 22:32

Nice regions, but hard to visit even with reduced view range.
Not sure why udp is at 400kb/s all the time there and physics so high

try setting
UpdatePrioritizationScheme = SimpleAngularDistance
ObjectsCullingByDistance = true

on opensim.ini [InterestManagement]
(0033437)
SnootsDwagon (reporter)
2018-11-07 08:13

Thanks Ubit!
(0033438)
SnootsDwagon (reporter)
2018-11-07 08:36
edited on: 2018-11-07 09:43

I thought it good to post an illustration here of what is happening currently on Opensim in regard to texture rezzing:

When an avatar is on a region and we tell it to walk forward, we are making an "on demand" request of the system. We expect the avatar to start walking immediately, not sometime in the future. When we turn the avatar in a circle, we expect it to immediately start turning in a circle. If we tell it to fly, we expect it to immediately start flying. When we alt-click on an object to zoom in, we expect to zoom in now.

In all of these activities, we expect immediate response from the server. We expect these things to occur NOW, not 16 seconds from now. If we try to walk and that doesn't happen for 5, 10, 15 or more seconds, that is what we call "lag"... the system not responding properly to "on demand" user actions.

THE VERY SAME holds true for texture rezzing. When we click on a vendor, click a slideshow item, edit a prim, zoom in close on a surface, double click a texture from inventory, or drag a texture from inventory to a prim... we are performing an "on demand" action. We rightfully expect immediate response from the server, namely... the texture loading being given as much priority as walking, and that texture appearing NOW, not 16 seconds from now.

The system is not prioritizing on-demand texture use. It should be doing so. That is the elusive "bottleneck" that is the core of this issue: simple failure to prioritize on-demand avatar actions in regard to texture rezzing.

Of course there may be other bottlenecks, but fix that priority and my guess is you'll have fixed the vast majority of the problem-- at least to the point no one will complain about it any more. Personally, I can handle a texture rezzing in 2 to 3 seconds. 13 to 16 seconds... not so much. That adds up to a whole lot of wasted user time when rezzing multiple textures (such as when trying to build or shop).

*High-prioritize texture on-demand actions*. Respectfully, that appears to be the initial solution-step to this problem. Fix that, and minor texture bottlenecks can be hunted down and improved later. Prioritizing on-demand texture rezzing will take care of the vast majority of the texture issue and put a closure on this Mantis.

(As a side-note: the same can be said for use of gestures or sounds. How many times have we tried to use gestures, only to have them come through 30 seconds later? How many times have we tried to listen to a music player only to have it "lag" and skip music clips? Educated guess: same exact issue-- lack of on-demand prioritization. This logical concept is affecting many process, grid-wide, metaverse-wide. Fix on-demand prioritization-- and fix some of the oldest and most major problems in the metaverse.)

(0033442)
UbitUmarov (administrator)
2018-11-07 10:24

Same as mantis 8404. Overall system performance issues, most viewer side, in particular with overloaded regions.

- Issue History
Date Modified Username Field Change
2018-10-17 10:56 SnootsDwagon New Issue
2018-10-18 01:18 Sheera Khan Note Added: 0033172
2018-10-18 09:44 Luisillo_Contepomi Note Added: 0033174
2018-10-18 10:15 SnootsDwagon Note Added: 0033175
2018-10-18 10:21 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 10:23 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 10:24 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 10:33 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 10:37 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 10:39 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 10:39 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 11:06 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 11:07 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 11:09 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 11:09 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-18 11:10 SnootsDwagon Note Edited: 0033175 View Revisions
2018-10-19 07:04 Luisillo_Contepomi Note Added: 0033180
2018-10-19 07:06 Luisillo_Contepomi Note Edited: 0033180 View Revisions
2018-10-19 08:40 SnootsDwagon Note Added: 0033181
2018-10-19 08:41 SnootsDwagon Note Edited: 0033181 View Revisions
2018-10-19 08:43 SnootsDwagon Note Edited: 0033181 View Revisions
2018-10-19 08:45 SnootsDwagon Note Edited: 0033181 View Revisions
2018-10-19 12:57 Garmin Kawaguichi Note Added: 0033196
2018-10-19 13:30 SnootsDwagon Note Added: 0033197
2018-10-26 10:39 Sheera Khan Relationship added related to 0008304
2018-10-26 11:02 SnootsDwagon Note Deleted: 0033197
2018-10-26 11:10 SnootsDwagon Note Added: 0033268
2018-10-26 11:12 SnootsDwagon Note Edited: 0033268 View Revisions
2018-10-26 11:14 SnootsDwagon Note Edited: 0033268 View Revisions
2018-10-26 11:16 SnootsDwagon Note Edited: 0033268 View Revisions
2018-10-26 11:20 SnootsDwagon Note Edited: 0033268 View Revisions
2018-10-27 01:05 aiaustin Note Added: 0033273
2018-10-27 01:06 aiaustin Note Edited: 0033273 View Revisions
2018-10-27 01:18 aiaustin Note Edited: 0033273 View Revisions
2018-10-27 01:18 aiaustin Note Edited: 0033273 View Revisions
2018-10-27 01:19 aiaustin Note Edited: 0033273 View Revisions
2018-10-27 01:20 aiaustin Note Edited: 0033273 View Revisions
2018-10-27 01:21 aiaustin Note Edited: 0033273 View Revisions
2018-10-27 06:43 SnootsDwagon Note Added: 0033275
2018-10-27 07:19 aiaustin Note Added: 0033276
2018-10-27 07:22 aiaustin Note Edited: 0033276 View Revisions
2018-10-27 07:23 aiaustin Note Edited: 0033276 View Revisions
2018-10-27 07:24 aiaustin Note Edited: 0033276 View Revisions
2018-10-27 10:36 SnootsDwagon Note Added: 0033277
2018-10-27 11:00 FreakyTech Note Added: 0033278
2018-10-27 11:01 FreakyTech Note Edited: 0033278 View Revisions
2018-10-27 13:13 SnootsDwagon Note Edited: 0033277 View Revisions
2018-10-29 04:36 aiaustin Additional Information Updated View Revisions
2018-11-02 04:38 aiaustin Note Added: 0033381
2018-11-05 12:36 SnootsDwagon Note Added: 0033403
2018-11-05 12:36 SnootsDwagon Note Edited: 0033403 View Revisions
2018-11-05 12:36 SnootsDwagon Note Edited: 0033403 View Revisions
2018-11-05 12:38 SnootsDwagon Note Edited: 0033403 View Revisions
2018-11-05 12:39 SnootsDwagon Note Edited: 0033403 View Revisions
2018-11-05 12:40 SnootsDwagon Note Edited: 0033403 View Revisions
2018-11-05 13:37 BillBlight Note Added: 0033404
2018-11-05 14:11 SnootsDwagon Note Added: 0033405
2018-11-05 14:14 SnootsDwagon Note Edited: 0033405 View Revisions
2018-11-05 14:16 SnootsDwagon Note Edited: 0033405 View Revisions
2018-11-05 14:17 SnootsDwagon Note Edited: 0033405 View Revisions
2018-11-05 14:19 SnootsDwagon Note Edited: 0033405 View Revisions
2018-11-05 14:21 SnootsDwagon Note Edited: 0033405 View Revisions
2018-11-05 14:22 SnootsDwagon Note Edited: 0033405 View Revisions
2018-11-05 14:23 BillBlight Note Added: 0033406
2018-11-05 14:26 SnootsDwagon Note Edited: 0033405 View Revisions
2018-11-05 14:29 BillBlight Note Edited: 0033406 View Revisions
2018-11-05 14:32 SnootsDwagon Note Added: 0033407
2018-11-05 14:34 SnootsDwagon Note Edited: 0033407 View Revisions
2018-11-05 14:36 SnootsDwagon Note Edited: 0033405 View Revisions
2018-11-05 14:37 SnootsDwagon Note Edited: 0033407 View Revisions
2018-11-05 14:43 BillBlight Note Added: 0033408
2018-11-05 14:59 SnootsDwagon Note Added: 0033409
2018-11-05 15:01 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:04 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:09 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:09 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:11 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:16 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:17 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:18 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:19 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:20 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:20 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:21 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:22 SnootsDwagon Note View State: 0033409: private
2018-11-05 15:23 SnootsDwagon Note View State: 0033409: public
2018-11-05 15:35 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:35 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:38 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:39 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:41 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:55 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:56 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 15:58 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 16:08 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 16:09 UbitUmarov Note Added: 0033410
2018-11-05 16:11 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-05 17:16 SnootsDwagon Note Added: 0033411
2018-11-05 17:18 SnootsDwagon Note Edited: 0033411 View Revisions
2018-11-05 17:22 SnootsDwagon Note Added: 0033412
2018-11-05 17:25 SnootsDwagon Note Edited: 0033412 View Revisions
2018-11-05 17:26 SnootsDwagon Note Edited: 0033412 View Revisions
2018-11-05 17:26 SnootsDwagon Note Edited: 0033412 View Revisions
2018-11-05 17:27 SnootsDwagon Note Edited: 0033412 View Revisions
2018-11-05 17:29 SnootsDwagon Note Edited: 0033412 View Revisions
2018-11-05 17:29 SnootsDwagon Note Edited: 0033412 View Revisions
2018-11-05 19:19 SnootsDwagon Note Edited: 0033411 View Revisions
2018-11-05 19:20 SnootsDwagon Note Edited: 0033412 View Revisions
2018-11-06 05:48 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-06 05:49 SnootsDwagon Note Edited: 0033409 View Revisions
2018-11-06 05:50 SnootsDwagon Note Edited: 0033412 View Revisions
2018-11-06 05:55 SnootsDwagon Note Edited: 0033412 View Revisions
2018-11-06 05:56 SnootsDwagon Note Edited: 0033412 View Revisions
2018-11-06 06:40 aiaustin Note Added: 0033414
2018-11-06 06:41 aiaustin Note Edited: 0033414 View Revisions
2018-11-06 06:43 aiaustin Note Edited: 0033414 View Revisions
2018-11-06 06:49 aiaustin Note Edited: 0033414 View Revisions
2018-11-06 07:01 aiaustin Note Edited: 0033414 View Revisions
2018-11-06 07:02 aiaustin Note Edited: 0033414 View Revisions
2018-11-06 07:23 aiaustin Note Edited: 0033414 View Revisions
2018-11-06 07:31 aiaustin Note Edited: 0033414 View Revisions
2018-11-06 15:30 SnootsDwagon Note Added: 0033423
2018-11-06 20:30 SnootsDwagon Note Added: 0033434
2018-11-06 21:21 SnootsDwagon Note Added: 0033435
2018-11-06 21:22 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:24 SnootsDwagon Note Edited: 0033434 View Revisions
2018-11-06 21:27 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:29 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:30 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:30 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:33 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:34 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:37 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:38 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:39 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:39 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 21:44 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-06 22:32 UbitUmarov Note Added: 0033436
2018-11-07 08:13 SnootsDwagon Note Added: 0033437
2018-11-07 08:19 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-07 08:21 SnootsDwagon Note Edited: 0033435 View Revisions
2018-11-07 08:23 SnootsDwagon Note Edited: 0033434 View Revisions
2018-11-07 08:36 SnootsDwagon Note Added: 0033438
2018-11-07 08:37 SnootsDwagon Note Edited: 0033438 View Revisions
2018-11-07 08:39 SnootsDwagon Note Edited: 0033438 View Revisions
2018-11-07 08:42 SnootsDwagon Note Edited: 0033438 View Revisions
2018-11-07 08:42 SnootsDwagon Note Edited: 0033438 View Revisions
2018-11-07 09:43 SnootsDwagon Note Edited: 0033438 View Revisions
2018-11-07 10:11 UbitUmarov Relationship deleted related to 0008304
2018-11-07 10:21 UbitUmarov Relationship added related to 0008404
2018-11-07 10:24 UbitUmarov Note Added: 0033442
2018-11-07 10:24 UbitUmarov Status new => closed
2018-11-07 10:24 UbitUmarov Assigned To => UbitUmarov
2018-11-07 10:24 UbitUmarov Resolution open => won't fix


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker