Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008391opensim[GRID] Grid Servicepublic2018-10-06 16:062018-10-09 01:45
Assigned To 
PlatformOSOS Version
Product Version0.9.0.1 
Target VersionFixed in Version 
Summary0008391: Object Contents failing to load promptly
DescriptionAccessing the Contents of an object takes an extremely (and frustratingly) long time. Updating an object with 300 to 500 Contents can take 20 minutes or longer. This affects items such as texture sorters, music players, inventory backup boxes and more.
Additional InformationThis problem hampers building, slows down production and causes creators problems every day. Quote from long-time virtual tech Balpien Hammerer on "Opensim Virtual", which may be of help in fixing this major problem:

"I can do a MySQL query of 50,000 prims and have the query done in 1/10th of a second. There are two major bottlenecks. One is that OpenSIM's Robust's httpserver is anemic at best. It's a long known problem with OpenSim. So, if the asset fetches are coming form Robust it will be slow. InWorldz fixed a lot of that. There is also work underway on v0.9.x to fix that problem.

The second problem is what is being described that object create messages to the viewer or editing an object is very slow if it has a lot of prims (the linkset is large). The primary cause of that one is the fake TCP Linden Lab created decades ago that uses LLUDP. Object create and editing messages to the viewer are very slow because of the teensy window it uses. What InWorldz did (Tranq and I worked on that one) was to expand the window to batch up 8-16 packets at a time. It turns out the viewers know how to deal with that so the speed up can be implemented in the simulator.

Halcyon code is open source. You can look up the changes. Maybe someone can even take the change into OpenSim core! That would be very nice."
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (1 Region per Sim)
Physics EngineBulletSim
EnvironmentMono / Windows
Mono VersionNone
Attached Files

- Relationships
related to 0008202closed Edit object to edit contents i.e. scripts does not load or loads after LONG waiting time 
related to 0007904new Slow Object CONTENTS loading blocks building 

-  Notes
UbitUmarov (administrator)
2018-10-06 19:27

Think you maybe a little outdated, the possible speed ups where long introduced
But never expect very fast response, bc viewers protocol was just not made for large numbers.
BillBlight (developer)
2018-10-06 19:30

I tried to replicate this, while not as fast as I'd like it to be still only took between 15 and 40 seconds to open and item with 640 items in it ...

Seems to me if it was systemic code issue taking 20 minutes it would affect me just like affects others??

If it takes varying amounts of time, I'm sure that is based on traffic..
SnootsDwagon (reporter)
2018-10-06 20:56
edited on: 2018-10-06 21:17

500 items is not a "large number". If it is considered so, Opensim will never scale to potential.

"Between 15 and 40 seconds" is too long to open a simple database of 640 items. (15 seconds is barely tolerable. 40 is excessive.) If this is "based on traffic" then the asset servers and bandwidth are not up to system demands.

Folks, I was a system designer and coder in RL for 25 years before retiring. I well understand what is acceptable performance-- and what isn't. Ubit respectfully, if "possible speed ups were long introduced"... they were not sufficient. Such efforts did not succeed to the degree possible. More is needed. One should never believe everything has been done that can be done.

Whether this has been tackled before or not, the issue still exists. If we are to believe that the terribly slow Contents fetching that now exists is as fast as it can get-- that does not bode well for the future of Opensim.

In a competitive business environment, business managers and coders consider such things very seriously: they realize the problem exists, and they don't give up until it is fixed, completely. The business owner tells IT, "This needs fixed. Fix it". And IT fixes it, one way or another. That is real life, for-profit paid coding. Here we have volunteers, and those volunteers are appreciated. But if these problems continue and people prosper the idea "it can't get any better than this"... then it definitely won't get any better.

When we click the CONTENTS tab of an object, that inventory should load faster than greased lightning. If it doesn't, something is definitely wrong. No excuses, no denials. This is a MAJOR bug.

"This is how it is" will not make Opensim better. Progress comes from fixing things that need fixed-- and fixing them so they work right, at maximum potential.

Let there be no doubt: there IS a bottleneck here. Where that bottleneck is, how it affects different people, how to fix it-- that needs to be determined. I'm reporting this with 25 years of professional coding and 18 years of HEAVY virtual world experience behind me. This report can either be acknowledged and the problem tackled, or Opensim can deny such reports and continue limping along because problems like this fail to get fixed.

Before making that decision-- ask Balpien Hammerer about the time he and I reported a texture bottleneck to Linden Lab (first in 2005 and again in 2007) and what ensued over the next FIVE YEARS as that company continued to deny the existence of such a bottleneck. At the end of that 5 years they suddenly "discovered" the very bottleneck we'd reported years before. FIVE YEARS that problem remained, daily damaging user retention until Linden Lab figured out we knew what we were talking about in the first place-- and fixed that major texture loading issue.

If they had listened to my first report in 2005... if they'd have taken that report seriously enough to run some tests and confirm the reality... their system would have worked a lot better than it did prior to 2010. Instead they denied the problem existed and it continued to damage Second Life for five years after it was initially reported.

There IS a bottleneck here. Now, whether that bottleneck exists in Opensim code, in the Robust asset server, in Firestorm or all of the above-- that needs to be determined. But that bottleneck is there as sure as I'm typing this sentence.

THE CORRECT BEHAVIOR should be: Edit an object, click the CONTENTS tab, and that inventory pops up fast as a blink. The database engine is fully capable of handling such action, especially with a mere 500 items. If that action is not occurring-- it is because one or more processes are not functioning as they should function.

The bottleneck that Balpien and I reported to Linden Lab was serious enough to actually constitute an internal DDoS attack on their own system. I would not be a bit surprised if something similar is going on here.

The upside: if someone locates THIS bottleneck... you just might discover a logic issue that is affecting the entirety of Opensim code-- and wind up fixing a whole lot more than just object Content loading.

SnootsDwagon (reporter)
2018-10-06 22:24
edited on: 2018-10-07 10:34

As an interesting experiment I stayed up past bedtime tonight (BAD DWAGON!) and ran some tests on SL. Here are the results:

Single plain-prim box filed with 520 textures. Computer is a newgen i5 3.8ghz quad core, GeForce 1050 2 gig graphics card, 240gig SSD hard drive running at 6gbs, and a 25mb cable internet feed (more than sufficient). Not a slow system.

Freshly-installed SL Viewer: averaged 5 seconds to 27 seconds, and much in between. (Note: this in itself tells us that even on SL this function doesn't work properly or consistently. I consider the 2 or 3 instances of 5 seconds to be the *accurate, possible, to-be-expected* speed on such a process. The other speeds indicate bottleneck issues even on Second Life-- which should come as a surprise to no one.)

Freshly, clean-installed Firestorm viewer on the same box on SL: Fastest retrieval was 35 seconds. I stopped tracking when it went past 3 minutes.

This indicates at least part of the bottleneck is in the Firestorm Viewer itself. It is by evidence slower than the SL Viewer. Fastest times: SL Viewer 5 seconds compared to Firestorm 35 seconds. Slowest time: SL 27 seconds compared to... Firestorm user gets tired of waiting.

Just an added piece of evidence. I'll try to gather more data tomorrow.

BillBlight (developer)
2018-10-07 09:44

Since it seems like even with just this evidence that it is viewer related, really should open a jira with firestorm ...
SnootsDwagon (reporter)
2018-10-07 10:24
edited on: 2018-10-07 10:57

I tried that some time ago Watcher, but there was denial the issue exists (and to be honest, a bit of attitude) and they closed the report as "unsubstantiated". It was obvious nobody bothered to test and see if the report was correct; they wanted everyone else to run tests for them-- and when we did they ignored the data that was provided. Users are not expected to have the technical expertise to perform debug testing.

It was suggested above this be tested on the Inworldz viewer. Are those who made such suggestion unable to run such tests themselves, for the good of the grid?

I agree with you that the evidence points to Firestorm at least partially, but I think it would be a mistake to point the finger solely at the viewer. The reason being: in all tests, the findings were not consistent. Even on Second Life using the SL viewer, the times differed between 5 seconds and 27 seconds. In my opinion the time on retrieving 500 items should be about 5 seconds (10 at the outside) consistently... which means that unless the same bottleneck exists in both the SL and Firestorm viewers (which is quite possible), the entire process of both viewer and server-side needs to be thoroughly examined, by all parties involved in code debugging and maintenance. Firestorm and Opensim may even need to cooperate-- since it is evident Firestorm is having a problem on the SL platform as well.

UbitUmarov (administrator)
2018-10-08 09:46

This days it seems that to have fs look to a jira you can not use the word OpenSim anywhere, or it will automatically rejected. Possible with a not so nice message.
SnootsDwagon (reporter)
2018-10-08 13:00
edited on: 2018-10-08 15:15

That seems to have been my experience Ubit. I find such things unfortunate.

I am reminded of the statement: "United we stand, divided we fall." I

It seems to me that if everyone in the Opensim / Firestorm community organized and contributed to a centralized server / viewer core systems without all these branches and individual releases that just muck up the works-- both systems would be a lot more vibrant. After 11+ years, the foundation could be more stable, bugs fixed pretty-durn-quick, and Opensim *could* have left SL behind technically. Instead, it's still a "runner up" tech-wise.

UNITED we stand. Divided... not so much. My thoughts: if Firestorm is as you state refusing to cooperate with Opensim, high time the Opensim community developed its own viewer and put a permanent end to that problem. The sooner we cut the SL apron strings... the better. Opensim needs to stand fully on its own rather than trying to mimic everything Linden Lab slams upon the community.

(This public service announcement brought to you by Opinionated Dwagons LLC and may not necessarily be the opinion of this Mantis or the community... though it should be.) Mwahahahaahaa...

BillBlight (developer)
2018-10-08 13:12
edited on: 2018-10-08 13:12

The following statements reflect personal opinions and in no way are any official statement from anyone but ME.

I don't disagree with you SnootsDwagoon ..

Here is the issue, the same people that scream "We Wan't NEW things" are the same people that scream, "OMG!!! YOU CHANGED SOMETHING, PUT IT BACK!!". It seems the community does not like or really want change they are complacent and only a few "Opensimulator Geeks" really want to work toward not being the "SL Knockoff".

They would rather their stuff be "broken" and "braindead" than have to put any effort into doing things the right way, sadly this is the general thought process of the overall community. "We don't care if it's broken as long as I can get my stuff and go dance in a virtual club"

aiaustin (developer)
2018-10-09 01:45

Folks, I just have to say that I have always found the Firestorm viewer developers to be helpful in identifying issues that may relate to OpenSim and do indeed make changes when issues are identified that are viewer side. But they do think some issues could be addressed by the OpenSim community when the server is identified as the likely cause and may make such statements in the Firestorm JIRA. I suggest we don't take them as negativity to OpenSim.

- Issue History
Date Modified Username Field Change
2018-10-06 16:06 SnootsDwagon New Issue
2018-10-06 19:27 UbitUmarov Note Added: 0033131
2018-10-06 19:30 BillBlight Note Added: 0033132
2018-10-06 19:32 UbitUmarov Relationship added related to 0008202
2018-10-06 19:34 UbitUmarov Relationship added related to 0007904
2018-10-06 20:56 SnootsDwagon Note Added: 0033133
2018-10-06 20:57 SnootsDwagon Note Edited: 0033133 View Revisions
2018-10-06 21:03 SnootsDwagon Note Edited: 0033133 View Revisions
2018-10-06 21:04 SnootsDwagon Note Edited: 0033133 View Revisions
2018-10-06 21:09 SnootsDwagon Note Edited: 0033133 View Revisions
2018-10-06 21:17 SnootsDwagon Note Edited: 0033133 View Revisions
2018-10-06 22:24 SnootsDwagon Note Added: 0033134
2018-10-07 09:44 BillBlight Note Added: 0033141
2018-10-07 10:24 SnootsDwagon Note Added: 0033143
2018-10-07 10:32 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:34 SnootsDwagon Note Edited: 0033134 View Revisions
2018-10-07 10:36 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:36 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:36 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:48 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:49 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:57 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-08 09:46 UbitUmarov Note Added: 0033150
2018-10-08 13:00 SnootsDwagon Note Added: 0033151
2018-10-08 13:01 SnootsDwagon Note Edited: 0033151 View Revisions
2018-10-08 13:03 SnootsDwagon Note Edited: 0033151 View Revisions
2018-10-08 13:04 SnootsDwagon Note Edited: 0033151 View Revisions
2018-10-08 13:12 BillBlight Note Added: 0033152
2018-10-08 13:12 BillBlight Note Edited: 0033152 View Revisions
2018-10-08 15:15 SnootsDwagon Note Edited: 0033151 View Revisions
2018-10-09 01:45 aiaustin Note Added: 0033153

Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker