Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008391opensim[GRID] Grid Servicepublic2018-10-06 16:062019-06-11 14:34
ReporterSnootsDwagon 
Assigned To 
PriorityurgentSeveritymajorReproducibilityalways
StatusnewResolutionopen 
PlatformOSOS Version
Product Version0.9.0.1 
Target VersionFixed in Version 
Summary0008391: Object Contents failing to load promptly
DescriptionAccessing the Contents of an object takes an extremely (and frustratingly) long time. Updating an object with 300 to 500 Contents can take 20 minutes or longer. This affects items such as texture sorters, music players, inventory backup boxes and more.
Additional InformationThis problem hampers building, slows down production and causes creators problems every day. Quote from long-time virtual tech Balpien Hammerer on "Opensim Virtual", which may be of help in fixing this major problem:

"I can do a MySQL query of 50,000 prims and have the query done in 1/10th of a second. There are two major bottlenecks. One is that OpenSIM's Robust's httpserver is anemic at best. It's a long known problem with OpenSim. So, if the asset fetches are coming form Robust it will be slow. InWorldz fixed a lot of that. There is also work underway on v0.9.x to fix that problem.

The second problem is what is being described that object create messages to the viewer or editing an object is very slow if it has a lot of prims (the linkset is large). The primary cause of that one is the fake TCP Linden Lab created decades ago that uses LLUDP. Object create and editing messages to the viewer are very slow because of the teensy window it uses. What InWorldz did (Tranq and I worked on that one) was to expand the window to batch up 8-16 packets at a time. It turns out the viewers know how to deal with that so the speed up can be implemented in the simulator.

Halcyon code is open source. You can look up the changes. Maybe someone can even take the change into OpenSim core! That would be very nice."
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (1 Region per Sim)
Physics EngineBulletSim
EnvironmentMono / Windows
Mono VersionNone
ViewerFirestorm
Attached Files

- Relationships
related to 0008202closed Edit object to edit contents i.e. scripts does not load or loads after LONG waiting time 
related to 0007904new Slow Object CONTENTS loading blocks building 

-  Notes
(0033131)
UbitUmarov (administrator)
2018-10-06 19:27

Think you maybe a little outdated, the possible speed ups where long introduced
But never expect very fast response, bc viewers protocol was just not made for large numbers.
(0033132)
BillBlight (developer)
2018-10-06 19:30

I tried to replicate this, while not as fast as I'd like it to be still only took between 15 and 40 seconds to open and item with 640 items in it ...

Seems to me if it was systemic code issue taking 20 minutes it would affect me just like affects others??

If it takes varying amounts of time, I'm sure that is based on traffic..
(0033133)
SnootsDwagon (reporter)
2018-10-06 20:56
edited on: 2018-10-06 21:17

500 items is not a "large number". If it is considered so, Opensim will never scale to potential.

"Between 15 and 40 seconds" is too long to open a simple database of 640 items. (15 seconds is barely tolerable. 40 is excessive.) If this is "based on traffic" then the asset servers and bandwidth are not up to system demands.

Folks, I was a system designer and coder in RL for 25 years before retiring. I well understand what is acceptable performance-- and what isn't. Ubit respectfully, if "possible speed ups were long introduced"... they were not sufficient. Such efforts did not succeed to the degree possible. More is needed. One should never believe everything has been done that can be done.

Whether this has been tackled before or not, the issue still exists. If we are to believe that the terribly slow Contents fetching that now exists is as fast as it can get-- that does not bode well for the future of Opensim.

In a competitive business environment, business managers and coders consider such things very seriously: they realize the problem exists, and they don't give up until it is fixed, completely. The business owner tells IT, "This needs fixed. Fix it". And IT fixes it, one way or another. That is real life, for-profit paid coding. Here we have volunteers, and those volunteers are appreciated. But if these problems continue and people prosper the idea "it can't get any better than this"... then it definitely won't get any better.

When we click the CONTENTS tab of an object, that inventory should load faster than greased lightning. If it doesn't, something is definitely wrong. No excuses, no denials. This is a MAJOR bug.

"This is how it is" will not make Opensim better. Progress comes from fixing things that need fixed-- and fixing them so they work right, at maximum potential.

Let there be no doubt: there IS a bottleneck here. Where that bottleneck is, how it affects different people, how to fix it-- that needs to be determined. I'm reporting this with 25 years of professional coding and 18 years of HEAVY virtual world experience behind me. This report can either be acknowledged and the problem tackled, or Opensim can deny such reports and continue limping along because problems like this fail to get fixed.

Before making that decision-- ask Balpien Hammerer about the time he and I reported a texture bottleneck to Linden Lab (first in 2005 and again in 2007) and what ensued over the next FIVE YEARS as that company continued to deny the existence of such a bottleneck. At the end of that 5 years they suddenly "discovered" the very bottleneck we'd reported years before. FIVE YEARS that problem remained, daily damaging user retention until Linden Lab figured out we knew what we were talking about in the first place-- and fixed that major texture loading issue.

If they had listened to my first report in 2005... if they'd have taken that report seriously enough to run some tests and confirm the reality... their system would have worked a lot better than it did prior to 2010. Instead they denied the problem existed and it continued to damage Second Life for five years after it was initially reported.

There IS a bottleneck here. Now, whether that bottleneck exists in Opensim code, in the Robust asset server, in Firestorm or all of the above-- that needs to be determined. But that bottleneck is there as sure as I'm typing this sentence.

THE CORRECT BEHAVIOR should be: Edit an object, click the CONTENTS tab, and that inventory pops up fast as a blink. The database engine is fully capable of handling such action, especially with a mere 500 items. If that action is not occurring-- it is because one or more processes are not functioning as they should function.

The bottleneck that Balpien and I reported to Linden Lab was serious enough to actually constitute an internal DDoS attack on their own system. I would not be a bit surprised if something similar is going on here.

The upside: if someone locates THIS bottleneck... you just might discover a logic issue that is affecting the entirety of Opensim code-- and wind up fixing a whole lot more than just object Content loading.

(0033134)
SnootsDwagon (reporter)
2018-10-06 22:24
edited on: 2018-10-07 10:34

As an interesting experiment I stayed up past bedtime tonight (BAD DWAGON!) and ran some tests on SL. Here are the results:

Single plain-prim box filed with 520 textures. Computer is a newgen i5 3.8ghz quad core, GeForce 1050 2 gig graphics card, 240gig SSD hard drive running at 6gbs, and a 25mb cable internet feed (more than sufficient). Not a slow system.

Freshly-installed SL Viewer: averaged 5 seconds to 27 seconds, and much in between. (Note: this in itself tells us that even on SL this function doesn't work properly or consistently. I consider the 2 or 3 instances of 5 seconds to be the *accurate, possible, to-be-expected* speed on such a process. The other speeds indicate bottleneck issues even on Second Life-- which should come as a surprise to no one.)

Freshly, clean-installed Firestorm viewer on the same box on SL: Fastest retrieval was 35 seconds. I stopped tracking when it went past 3 minutes.

This indicates at least part of the bottleneck is in the Firestorm Viewer itself. It is by evidence slower than the SL Viewer. Fastest times: SL Viewer 5 seconds compared to Firestorm 35 seconds. Slowest time: SL 27 seconds compared to... Firestorm user gets tired of waiting.

Just an added piece of evidence. I'll try to gather more data tomorrow.

(0033141)
BillBlight (developer)
2018-10-07 09:44

Since it seems like even with just this evidence that it is viewer related, really should open a jira with firestorm ...
(0033143)
SnootsDwagon (reporter)
2018-10-07 10:24
edited on: 2018-10-07 10:57

I tried that some time ago Watcher, but there was denial the issue exists (and to be honest, a bit of attitude) and they closed the report as "unsubstantiated". It was obvious nobody bothered to test and see if the report was correct; they wanted everyone else to run tests for them-- and when we did they ignored the data that was provided. Users are not expected to have the technical expertise to perform debug testing.

It was suggested above this be tested on the Inworldz viewer. Are those who made such suggestion unable to run such tests themselves, for the good of the grid?

I agree with you that the evidence points to Firestorm at least partially, but I think it would be a mistake to point the finger solely at the viewer. The reason being: in all tests, the findings were not consistent. Even on Second Life using the SL viewer, the times differed between 5 seconds and 27 seconds. In my opinion the time on retrieving 500 items should be about 5 seconds (10 at the outside) consistently... which means that unless the same bottleneck exists in both the SL and Firestorm viewers (which is quite possible), the entire process of both viewer and server-side needs to be thoroughly examined, by all parties involved in code debugging and maintenance. Firestorm and Opensim may even need to cooperate-- since it is evident Firestorm is having a problem on the SL platform as well.

(0033150)
UbitUmarov (administrator)
2018-10-08 09:46

This days it seems that to have fs look to a jira you can not use the word OpenSim anywhere, or it will automatically rejected. Possible with a not so nice message.
(0033151)
SnootsDwagon (reporter)
2018-10-08 13:00
edited on: 2018-10-08 15:15

That seems to have been my experience Ubit. I find such things unfortunate.

I am reminded of the statement: "United we stand, divided we fall." I

It seems to me that if everyone in the Opensim / Firestorm community organized and contributed to a centralized server / viewer core systems without all these branches and individual releases that just muck up the works-- both systems would be a lot more vibrant. After 11+ years, the foundation could be more stable, bugs fixed pretty-durn-quick, and Opensim *could* have left SL behind technically. Instead, it's still a "runner up" tech-wise.

UNITED we stand. Divided... not so much. My thoughts: if Firestorm is as you state refusing to cooperate with Opensim, high time the Opensim community developed its own viewer and put a permanent end to that problem. The sooner we cut the SL apron strings... the better. Opensim needs to stand fully on its own rather than trying to mimic everything Linden Lab slams upon the community.

(This public service announcement brought to you by Opinionated Dwagons LLC and may not necessarily be the opinion of this Mantis or the community... though it should be.) Mwahahahaahaa...

(0033152)
BillBlight (developer)
2018-10-08 13:12
edited on: 2018-10-08 13:12

The following statements reflect personal opinions and in no way are any official statement from anyone but ME.


I don't disagree with you SnootsDwagoon ..

Here is the issue, the same people that scream "We Wan't NEW things" are the same people that scream, "OMG!!! YOU CHANGED SOMETHING, PUT IT BACK!!". It seems the community does not like or really want change they are complacent and only a few "Opensimulator Geeks" really want to work toward not being the "SL Knockoff".

They would rather their stuff be "broken" and "braindead" than have to put any effort into doing things the right way, sadly this is the general thought process of the overall community. "We don't care if it's broken as long as I can get my stuff and go dance in a virtual club"

(0033153)
aiaustin (developer)
2018-10-09 01:45

Folks, I just have to say that I have always found the Firestorm viewer developers to be helpful in identifying issues that may relate to OpenSim and do indeed make changes when issues are identified that are viewer side. But they do think some issues could be addressed by the OpenSim community when the server is identified as the likely cause and may make such statements in the Firestorm JIRA. I suggest we don't take them as negativity to OpenSim.
(0035324)
beqjanus (reporter)
2019-06-07 16:25

Hi all,
I'm an FS dev and I've been looking at this issue but I cannot find any issue in the viewer side or any evidence to help me progress further.

Here is the scenario that I have tested.
I was given an object (a glitterball full of 750+ animations) on Littlefield. I am assured by Walter and Aine that said object will fail to load the contents or take "forever". I have edited the object many times now and while it is slow (about a minute or so) it has never yet failed.

To look further into the slowness I took a wire shark packet capture. I also timed the start to end that I saw "visually" on screen in the viewer.

The elapsed wallclock time was around 1 minute 20 seconds

The capture showed that the viewer and server were in constant conversation throughout that time and that the network completion coincided pretty much with the visual update. Furthermore, the interpacket latency while not lightning fast is not excessively slow, with the viewer ack to the preceding packet typically appear with 20-30ms.

The following image shows the start of the requests. It is filtered to remove all the other trafic and the highlighted column shows the latency between "displayed packets"

https://gyazo.com/a1f95935be09634563f88dd53fb4fec6 [^]

From that, I conclude that there is nothing inherently slow in the Firestorm processing. That's not to say that it couldn't be made faster but the bigger issue here would seem to be that it took a total of 968 packets or roughly 480 msg/ack exchanges with ~150ms transfer latency (which is pretty consistent with UK->US ping). The viewer is not a significant player in this, nor for that matter, is the server it all comes down to physics, geography and an outmoded protocol.

Let me know if I am testing the wrong things or looking in the wrong places.
(0035325)
SnootsDwagon (reporter)
2019-06-07 18:26
edited on: 2019-06-07 18:42

Greetings Begianus. Thank you for looking in to this. I know zero about the technical end of SL, Opensim or Firestorm so can't even begin to guess at what's going on behind the scenes. All I can offer is insight from a user:

Imagine one is a builder / scripter and working on a script... a good example being a song player or texture sorter. Whichever item, there are hundreds of Contents items. Every time an item is added, the script is changed, or the object accessed... there is that one minute 20 second delay.

From the user's standpoint, they're staring at a blank box-- in my experience from 45 seconds to 3 minutes or more. Every time a change is made. To a scripter updating a script, this can be maddening. To a creator trying to add inventory parts to an avatar clothing fatpack, to a music maker adding clips to the player... the time seems like forever. It can literally add and hour or more to development that should take minutes, proportionally more depending on the depth of the project.

As users we've no idea what causes the issue. Grid code? Script engine? Viewer? If it helps, I know this for a certainty: I filed a report about this on Inworldz (when they were still functioning) and they looked into it and somehow fixed it (so I'm suspecting grid code or script engine). The effect: Objects that were taking 1 minute, 3 minutes or more to load contents loaded exactly the same contents in 6 to 8 seconds (which is what I as a RL coder would expect when loading the names/uuids of database items). So they discovered the problem and did manage to get a reasonable level of speed.

Again, I'm speaking from "not behind the scenes", but realistically speaking, when we look at the contents of an object all we're seeing is the names of the assets. From my viewpoint that is a matter of accessing 100 to 500 (on an average) inventory items from the asset database and displaying their names on the screen. That should be able to be done lightning fast. I am sure there is a lot more going on behind the scenes, but what we're discussing above is basic data retrieval. There's no 3-D rezzing involved, we're not having to look at every single aspect of the asset. We just need to get its name, UUID and display the name in the contents box. Zap... there it should be.

I say this not only from RL experience, but from speaking with virtual techs, who assured me that yes, accessing such items is measured in milliseconds and that accessing a number of them should be extremely fast.

I have seen absolute proof that "extremely fast" is possible... because on occasion I have opened a "box" with over 300 items-- and those items displayed *almost instantly* and were ready to edit. It didn't happen consistently, but the fact that it happened at all proves it is possible to achieve such speeds. It also is strong indication of a serious logic or performance bottleneck.

Add to that the experiment I ran above, which I'll repeat here: compared a fresh-install of the SL viewer vs the Firestorem viewer, measuring the time it took to load contents in an object. Results: Fastest times: SL Viewer 5 seconds compared to Firestorm 35 seconds. Slowest time: SL 27 seconds compared to... Firestorm user gets tired of waiting.

Thus, empirical evidence not necessarily of a viewer malfunction, but definitely (at least) asset server communicating with viewer malfunction. For some reason the SL asset server was communicating with the SL viewer far more quickly than the Firestorm viewer. Or, to be honest, it could be almost totally a viewer bottleneck issue-- or some kind of odd handshake issue. Whatever the cause (which is what we're discussing here, the cause not the symptoms)... it was absolutely manifested in that repeatable test.

So that's why I filed the Mantis, because there is (from the user's viewpoint and from experiments we've run), evidently a major bottleneck somewhere. In my opinion, at no time should an object take longer than just a few seconds to load contents. It's basic database access with asset name display. That should be very, very fast, especially in an age when data access and transfer is measured in gigabytes per second.

Again, my thanks to you. :)

(0035326)
unregi (reporter)
2019-06-07 18:33

Hey beq, just wanted to say that you are awesome. Thanks for all the work you put into it.
(0035330)
beqjanus (reporter)
2019-06-08 05:20

SnootsDwagon, my evidence indicates that in OpenSim at least, the vast majority of the time is apparently spent in communication. You database analogy does not match the way that the legacy file xfer works.

It reminds me of my young son cleaning up his lego blocks. Picking them up one at a time and walking over to the box, placing it in and walking back to pick up the next.

What is needed is a grown-up to collect large handfuls of lego in her hands and make far fewer journeys. As it happens, the viewer can ask the region to use "big handfuls", and in fact I don't think the viewer really cares if the region sends big handfuls even when it doesn't ask for them.

I ran a test with "UseBigPackets" enabled (this is hardcoded at present) the opensim seems to ignore it and continued exactly as before, brick by brick.

https://gyazo.com/92185bfae1bf9a67efda8fac4a08d9a5 [^]

My hope was that we'd switch to a larger batch size if the viewer requested it but this does not appear to be supported. Can an OpenSim dev confirm this? I am using Littlefield, which I realise is older, but given that we struggle to keep OpenSim in focus at all I don't want to fragment into shards of OpenSim and would like to find a common solution if there is one.
(0035331)
SnootsDwagon (reporter)
2019-06-08 05:58
edited on: 2019-06-08 06:09

Sounds like you're on the right track Beqianus. Good analogy too with the Legos. What I'm not understanding as a user though, is that even if the system is picking up one Lego at a time... it should be doing so VERY quickly. Unless the asset server can't find its tail with both hands.

One has to realize we're talking as few as 50 content item and as high (on an average) of 500. If it takes 3 minutes to pull 500 items, that averages .36 seconds per data item-- which sounds fast but is in reality extremely slow. We should be seeing fire hose capability and instead are getting garden hose performance.

I agree with your concept that it seems handshaking between the viewer and server appears to the be issue. Realistically speaking, when the system asks the asset server for data, that data (even for 500 items) should come across in less than ONE SECOND. (That's ignoring Internet speeds, which is always a pothole.) The read speed of hard drives these days are blindingly fast at somewhere around 6 gpbs. (for the standard users out there, that's 6 billion bits per second). Taking MINUTES to retrieve that data... or even 20+ seconds, is technically speaking a serious "bottleneck"; something just isn't working right. Which is why I keep chirping on this: I know what the system is *capable* of doing... and it's not doing it. It's like owning a Porche with a top speed of 10mph. You know SOMETHING is wrong.

(0035332)
SnootsDwagon (reporter)
2019-06-08 06:18
edited on: 2019-06-08 06:21

Follow-up: "given that we struggle to keep OpenSim in focus at all I don't want to fragment into shards of OpenSim and would like to find a common solution if there is one. "

I can understand this sentiment. However, While Second Life is quite profitable and the "big dog" on the market, Opensim actually now out-ranks Second Life in number of active regions. So I appreciate the Firestorm team focuses on SL... but is that a realistic viewpoint in today's reality of virtual worlds?

When I was on Second Life we owned two regions and were paying some $700 a month for them, under Linden Lab's iron-thumb rule. When I was on Inworldz we had some 50+ regions and were paying that company some $2,000+ a month under their iron-lad rule.

On Open sim, we own 100 region, are serving them ourselves, have 110% control... for the cost of electricity (and the initial equipment). This indicates why so many people are migrating to Opensim-- and why it deserves significant support and consideration. I am rather surprised some corporations haven't jumped in there with some major pocketbook support.

The one thing we don't have control over is how Opensim is written, how the asset serer code is written, how the Viewer code is written. As users we have to accept what is provided. Which is why we rely on you generous folks to tackle these major bugs when we find them. We can find the bugs. We can identify the symptoms and sometimes (as in this case) consistently replicate them. What we can't do ourselves-- is get the coding up to snuff. That's where we rely on the devs... and why we sometimes get a bit wordy in the Mantis. ;D

We do a collective LOT of work out here creating virtual worlds. We need properly-functioning tools to do that work. Linden Lab got us off on the wrong foot with very poorly-written code. Firestorm and the Opensim teams are trying to fix that. I understand the need to keep supporting SL... but let's remember to not ignore the other big dog in the room. Collectively, Opensim has a great deal of user support.

(0035333)
aiaustin (developer)
2019-06-08 06:26
edited on: 2019-06-08 06:40

There were some changes in dev master by @UbitUmarov to sending content to the viewer in batches or compressed back around 2019-03-21 and soon after that changes to allow the viewer side object cache to be used. @UbitUmarov might be able to comment on whether these relate to the viewer “UseBigPackets” enabled flag. These changes are in place on grids on recent dev master code, such as recently updated regions on OSGrid (last updated 2019-05-14) but these commits are not yet on the stable release (last is 0.9.0.1 from 2018-09-03). Littlefield Grid is on 0.8.2.1 (from 2015-12-22).

@beqjanus, if you are able to run your packet tests on a recently updated grid, say on OSGrid, that might allow progress. Many thanks for this excellent testing @beqjanus and @SnootsDwagon.

(0035334)
FreakyTech (reporter)
2019-06-08 06:50
edited on: 2019-06-08 06:51

I made comparision tests and used aine's dancemaster as a reference data on osg's sandbox plaza and my fork.

test viewer fs 5.1.7 x64

The data set of prim inventory data is on

my fork: 480615 bytes
osg sandbox plaza: 492055 bytes

The parameters used for chunking transfer requests is:
my fork: 1000 bytes
osg sandbox plaza: 1024 bytes

required transfer blocks:
my fork: 480.615
osg sandbox plaza: 480.522

observed avg roundtrip time:
my fork: 70ms rtt => effective rate 14285 bytes per sec
osg sandbox plaza: 180ms rtt => effective rate 5688 bytes per sec

the 22s processing time observed has been excluded on osg sandbox plaza transfer request packet flow. During 0.8 times this happened earlier.
So there is actually 22s of delay generated by opensim on the test object.

so even that 0.9 variant used a few more bytes per blocks since it uses a more blown data stream it equals out the assumed increase of block size.

Assuming that part of the transfer rtt is generated by connectivity differences even using the two best numbers we end up with:

14628 bytes per second. However since 0.9 uses a larger data stream it is just not producing any improvement.


giving an excerpt of the first few items what is actually transfered:

    inv_object 0
    {
        obj_id a9446632-6153-41c0-8766-6a7ee35dda99
        parent_id 00000000-0000-0000-0000-000000000000
        type category
        name Contents|
    }
    inv_item 0
    {
        item_id 5471b9c7-eed8-470d-9ea1-6e82b0483444
        parent_id a9446632-6153-41c0-8766-6a7ee35dda99
        permissions 0
        {
            base_mask 0008e000
            owner_mask 0008e000
            group_mask 00000000
            everyone_mask 00008000
            next_owner_mask 0008e000
            creator_id 12a7488f-d08c-448e-93ff-6ae67500f709
            last_owner_id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
            group_id 00000000-0000-0000-0000-000000000000
            owner_id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
            group_owned 0
        }
        asset_id ead8725b-54bf-48a4-929a-7a43ae25e6c0
        type animatn
        inv_type animation
        flags 00080000
        sale_info 0
        {
            sale_type not
            sale_price 0
        }
        name stage19|
        desc Podium_19|
        creation_date 1416733113
    }
    inv_item 0
    {
        item_id 76f80759-8948-491d-8810-4327daa6d633
        parent_id a9446632-6153-41c0-8766-6a7ee35dda99
        permissions 0
        {
            base_mask 0008e000
            owner_mask 0008e000
            group_mask 00000000
            everyone_mask 00008000
            next_owner_mask 0008e000
            creator_id 12a7488f-d08c-448e-93ff-6ae67500f709
            last_owner_id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
            group_id 00000000-0000-0000-0000-000000000000
            owner_id xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
            group_owned 0
        }
        asset_id 4288bbe0-c1a0-46e3-a9b8-647cfd64873a
        type notecard
        inv_type notecard
        flags 00180000
        sale_info 0
        {
            sale_type not
            sale_price 0
        }
        name !!! VERSION 2.0 READ ME !!!|
        desc 2014-09-23 14:47:11 note card|
        creation_date 1452532591
    }


What really can be improved is the 22s preprocessing of the inventory data. It looks like an indicator for a string based generator instead of StringBuilder.

NB:Viewer object cache is not involved in object contents loading.

(0035335)
UbitUmarov (administrator)
2019-06-08 10:08
edited on: 2019-06-08 10:09

UseBigPackets option can not be safely supported across all platforms,
or worse, across all network paths

(0035336)
FreakyTech (reporter)
2019-06-08 10:31

UseBigPackets - If true, use large packet payloads (7680 bytes)
source: http://wiki.secondlife.com/wiki/RequestXfer [^]

That specified size of 7680 bytes is commonly blocked by intermediate routers these days due to fragmentation.

It will fail more often than not.

Adding MTU Path Discovery would be the only possibly to determine that reliably. An application packet parameter will not be able to determine that.
(0035337)
UbitUmarov (administrator)
2019-06-08 12:32

0.9xxx uses stringbuilder, of course...
I do see time in order of 20ms on "preprocess" of same object (possible other version) on a local LAN, including round trip. I consider this "normal" packets processing latency in fact I see mostly round trip.
the problem of this object contents transfer is really the protocol used.
And the protocol is exactly as Beq described, a little kid carrying just one or two legos (and can't carry more on his tiny hands as most routers will refuse also) at a time.
That means for me at osgrid around 200ms per lego, ie around 5kbps as freaky said
(0035338)
beqjanus (reporter)
2019-06-08 13:52
edited on: 2019-06-08 14:11

I tested the assertion that the LL viewer has an "immediate" response and this cannot be substantiated. In the tests that I ran using an object with 200+ items the LL viewer and the Firestorm viewer behaved almost identically, each results in 0000379:0000450 packets in the entire conversation with comparable inter-packet latencies.

As has been noted setting a larger buffer size is not going to work in the general case, but I wanted to test the capability as far as I can tell, the server just ignores the viewer's request and sends at the normal (safe) size.

I am happy to inquire about this with the lab and see if they are looking to upgrade this at all. My guess is that they won't, almost all large data transfers are now offloaded to CDN and use http, the majority of objects do not carry such large inventories and thus as a "problem", it will sit low on the list.

At present this seems at an impasse in the sense that it is what it is. A fix requires alterations on both side. I will try to do a capture on OSGrid to see how that fairs but I don't currently have any accounts or any inventory so if someone is able to assist in that I'd be most grateful.

There remains one concern, Aine drew my attention to this as one of the more "popular" problems and a priority to look at. But it has been said that often the viewer (FS) will never receive the updates and it will hang forever. I have not managed to achieve this at all, without a means to reproduce it,I cannot sensibly investigate and so, for now, this aspect remains an unknown.

(0035339)
UbitUmarov (administrator)
2019-06-08 14:21

Yes this is some concern since it makes building and testing very slow.
OpenSim does ignore the option UseBigPackets because the reasons already stated.
( 0.91 does send 1024 in place of 1000 just because I like to do things like blabla >>= 10 ;) )
(0035340)
FreakyTech (reporter)
2019-06-08 15:51

The real problem with the transfer protocol is that every fragment is transfered after the viewer has acknowledged the received one. This is a ping pong scheme. That is what makes the actual protocol handling slow.

TCP on the other case uses a sliding window approach. This means it is possible to send a serious of data packets and ideally getting acknowledges in time so that the stream of data does not stall.
(0035341)
UbitUmarov (administrator)
2019-06-08 16:04
edited on: 2019-06-08 16:15

yes, protocol is as Beq described, one little bit of legos at each time and get back to get more, I said at very first answer to this issue and in several other places.
0.91 does a bit better than that (but may make it a even worse if the net route is not that good, but will work)
No ideia why protocol is like that, other transfers via udp (textures) where different.
Any improvement will require a new protocol, in this case moving to a cap does make sense (more than other things now as cap). This can only be done as join devel by us and at least a viewer team, unless LL does change in a way we can use.
On top of protocol viewers shown other problems in past, like totally lock, hopefully now fixed. If my memory is not wrong they used to cache (good thing) them make a total mess with the cache (not good :) ) ??

(0035342)
UbitUmarov (administrator)
2019-06-08 16:06

in fact I may remove the 0.91 "improvement" to be again on the safe side
(0035343)
UbitUmarov (administrator)
2019-06-08 16:27

some of the problems I did remember:
https://jira.phoenixviewer.com/browse/FIRE-19738 [^]

currently I don't detect any caching
(0035344)
SnootsDwagon (reporter)
2019-06-08 16:52

(Scuse this lengthy post. I'm a user flying by the seat of my pants, but from RL experience know that some little thing pointed out may be an important key.)

Bequianus, I may not be able to assist on the inside-tech side, but I'll be happy to volunteer time helping you test anything you like on OSgrid. I have the inventory and objects ready to go and can follow your directions to retrieve whatever data you're looking for.

A few notes after reading the above:

* This report was filed some time ago (relatively speaking). Since it was first filed there have been some changes made to the code that (at least on OSgrid) was like a steroid shot. While it does still take some time to load large Contents inventories, I've not noticed the extremely long lag times that were present prior. It's not to the "ideal" level I'd like to see, but it's better than when I first filed this Mantis.

* There are still two facts that sit on the books: Inworldz did manage to get 300 to 500 items to load in 8 seconds. I have no idea how they did it, but it was with using the Firestorm Viewer. Wish I could give you more info on that, but they didn't reveal what they did to pull off that trick. I could live with 500 items loading in 8 seconds. If I had to guess on this: my guess would be they boosted the asset server code... or perhaps only loaded as much data as they needed to make the Contents box work. Namely: one doesn't have to load the ENTIRE asset data just to get the name and UUID. Whatever they did... it worked.

The second item is that test I ran comparing the SL Viewer with Firestorm. SL Viewer (much as I dislike its UI) worked with reasonably fast speeds, much faster than Firestorm (but not as fast as Inworldz). I probably should run the same identical test now to see if that situation still exists. (/me makes mental note)

Quoting FreakyTech above: "The real problem with the transfer protocol is that every fragment is transfered after the viewer has acknowledged the received one. This is a ping pong scheme. That is what makes the actual protocol handling slow. TCP on the other case uses a sliding window approach. This means it is possible to send a serious of data packets and ideally getting acknowledges in time so that the stream of data does not stall."

Freaky makes some valid points. I'm not sure how that can be addressed in the code, but as I mentioned some time back there, Balpien Hammerer and I conducted some tests way back on Second Life and discovered they were loading the same data repeatedly... up to 32 times... literally creating a DDoS attack on their own asset servers (imagine that bottleneck happening thousands of times a minute because of a 50,000 to 80,000 user concurrency).

* I run two 5x5 VARs on my own Opensim server tied in to OSgrid. This allows the ability to watch both instances on the same screen. Something that I have seen repeatedly (and informed OSgrid admin) is the same thing Balpien and I noticed on SL way back in 2007: the systems repeatedly sending the same data. It says so right on the screen. While a great deal of that is Greek to me, I can see yellow- and red-line areas during region loading where it states that information was re-sent repeatedly.

At the same time I have notice (especially recently) "Exceptionally long data retrieveal" notices, and those lines I did understand. It was taking the system an excessively long time to pull in a single asset from the asset server... 3 seconds, 5, seconds, often as long as 15 seconds to retrieve a single asset. Now that is during OAR loading. I don't know how that may apply to every-day operation, but my rule of thumb is: "A problem in one area can cascade into other areas. A problem fixed in one area might possibly fix all similar problems." This could especially apply to an asset server handshake or data retrieval routine.

As another example: for a very long time I have wondered by Teleporting was so regularly bad on OSgrid (and possibly the rest of Opensim. I don't know-- I don't get out much. :D). But when I set up my own server the reason for teleport failures became very obvious; I could see it right on the server screens. To put it in layman's terms:

Sender: Hi there! Here's an avatar I'm sending you.
Receiver: Thanks. Processing information now.
Sender: Good. I'm about done sending the data. Here's a bit more.
Receiver: Hmmm, having trouble receiving that. Hang on a sec, trying again.
Sender: THERE, ALL DONE! Have a nice day! (Hangs up the phone.)
Receiver: WAIT! I didn't get all the information. Trying to find a pen! Hold it! Well durn. Guess I'll just have to drop that avatar and let someone else worry about it.

As far as I'm able to tell among all the technobabble between the two instances, it's a handshake and communications protocol error. The Sender system transfers the avatar but doesn't wait and verify all the data has been received. The receiving unit loses track along the way, can't access the asset server, can't get the sender to resend the data, does not verify the avatar has been fully received, but the sender drops the feed anyway.

Considering that scenario on teleporting... imagine that same concept happening with any other asset on the grid, be it rezzing an object or (in the case of this Mantis), loading Contents in an object. Of course I doubt it's the same problem. Probably a different problem entirely. But I can SEE why teleports fail; it's right there on the screen, two regions on the same server screen trying to transfer an avatar between one another. It's not difficult to imagine similar communications issues during any kind of asset fetch / transfer.

Final item: it is heart-warming to see so many people jumping in here after this time and cooperating to try to get to the bottom of this. This kind of dedication-- especially unpaid-- encourages us users to assist the devs as much as we possibly can.

So again Beq, you need a tester, beep me. We'll set up a time and test like crazy. Same goes for Ubit, Freaky, aiaustin or any other dev who needs assist with testing. I don't like to test alone, blindly just because someone doesn't want to do it himself, but I am always happy to assist to help a dev collect real data. : )
(0035345)
SnootsDwagon (reporter)
2019-06-08 16:57
edited on: 2019-06-08 17:00

Ubit: "https://jira.phoenixviewer.com/browse/FIRE-19738 [^] [^]
currently I don't detect any caching"

YES Ubit! Bingo. That's another major issue. Access an item. Takes say, 30 seconds to load contents. Focus attention elsewhere. Go right back to the item... and it takes the same 30 seconds... again and again and again.

If there was a cache of some kind, surely would make major updating a large-contents item lots faster and easier. Mind you, it wouldn't solve whatever underlying issue is causing slow loading at this time. But it would help overall.

(0035346)
SnootsDwagon (reporter)
2019-06-08 17:06
edited on: 2019-06-08 17:07

CORRECTION. The point I made about Inworldz... incorrect. My memory just kicked in. They didn't succeed with that 8 second retrieval using Firestorm; I had to switch to the Inworldz Viewer to accomplish those speeds. So I ordinarily used Firestorm for everyday things, but when updating large-content items I would log out and switch to the Inworldz viewer. So my guess: they fixed Viewer to asset server communications.

Sorry for the misdirection. It's been a while and a LOT of water has gone under the bridge during that time... starting anew on OSgrid. ;D

(0035347)
UbitUmarov (administrator)
2019-06-08 17:14

I know what Inworldz did, their code is now opensource, and is not that different of what 0.91 does now, but it is a "hack" that may misbehave on bad network routes (will do it, but possible taking longer than vanilla protocol).

Teleports are different thing, they need communications between regions, grid services and viewer so never expect fast tps at osgrid between a region in rural Australia and rural Lithuania (just random examples).

Textures still another different issue :)
(0035348)
UbitUmarov (administrator)
2019-06-08 17:26

Yes to avoid the potential issues on bad routes, normal viewers would need change.
But just better another protocol...
I did not look for different protocol on this in IWz code.
(0035351)
FreakyTech (reporter)
2019-06-09 04:37
edited on: 2019-06-09 06:16

InWorldz added a sliding window acknowledge approach onto that protocol. That definitely requires viewer side support to have it work.

Interestingly there have not been changes to that code on inworldz viewer source code. A comparision with current linden lab viewer code revealed no adjustments related to that.

However, testing that revealed that FireStorm still is stepping on the brakes with that sliding window algorithm. So this concurs with Snoots observations.

(0035356)
UbitUmarov (administrator)
2019-06-09 12:05

in fact seems all viewers throttle to fixed(?) 100000 bits/s
(0035358)
SnootsDwagon (reporter)
2019-06-09 21:36
edited on: 2019-06-09 21:39

I ran a test on OSgrid tonight. Here are the results:

Object containing 689 Contents
Time to load on multiple tests: between 26 and 33 seconds

I don't know why the discrepancy, as one would think consistent results would be expected. But there are a LOT of variables on virtual worlds, so not unexpected.

So these readings are far better than when this Mantis was first filed. Someone did something right (and thanks to whoever tackled that).

Is it as good as it can get? No, since I have seen times as low as 6 to 8 seconds... and on very rare occasions almost instant (a second or so) which would be expected from a direct database read. If the system can fetch that many items in 1 to 2 seconds on rare occasions... from a layman's standpoint it means that for that moment... everything was in place and working as it should. Which leads us to believe it's possible to make everything in place the majority of the time (not always, nothing is perfect).

So, 26 to 33 seconds is far better than it used to be. But as has been pointed out by dev comments above (the Lego illustration), it can be significantly better.

I am thinking this asset-server fetch problem is one of the major issues with Opensim at this time. We see this when loading OAR files, when teleporting, when trying to rez an item. I am gratified and pleased to see our devs looking at this particular item now, because if you can get this fixed, that opens the way to fixing the other asset issues. Fix the asset issues, and we'll see such improvement on Opensim grids as no one ever thought possible. It won't eliminate all the bugs... but would significantly lessen their perceivable impact. We'd all love that. : )

Seems you folks are on the right path. I offer whatever encouragement possible to follow it through. The benefits to our worlds could be surprisingly significant.

(0035359)
FreakyTech (reporter)
2019-06-10 03:21
edited on: 2019-06-10 04:00

What you got wrong is that the protocol spec related to object contents loading is the one that is slow. However, that sits deep within shared code in viewers used for both SL/OS. Have not yet seen anyone wanting to completely split apart from SL in viewer terms.

What is weirder I have not found substantial evidence of changes on InWorldz viewer towards sliding window approach.
Which means the test set must have been different back then.

The object contents loading does not involve loading any assets.
It is just entries what the object inventory refers to.

It is not a direct database read at all.
The database is not even involved in retrieval of prim inventory.
Simply because that is done on startup of the simulator and the whole data is hold in memory.

When being able to replace the textual data into binary data that would provide a far higher gain on speed up. Because currently every entry has like 400 bytes that can be easily reduced to 100 bytes.

Substantial protocol changes deviating from LL upstream are only possible if OS completely declares all SL compatible viewers incompatible and builds an own one.

@Snoots if you can make LL change that, it would possibly be coming through that. Because most viewers still derive that very code part from LL upstream. Especially since most viewers target OpenSim as second priority.

Those 8 seconds you refer to misses actual reference data that was used back then. Anything else is just comparing apples with peaches.

There was once a similar thing mentioned by kitely and they used a way smaller test object for their tests.


Relevant data found via waybackmachine:

The description provides the actual reason why it actually did not deliver any speed up on the clubmaster inventory. Just 3 retransmits and the sliding window is 1. Effectively disabling the whole assumed improvement.

https://web.archive.org/web/20151128144139/http://wiki.inworldz.com/izwiki/index.php/R1800 [^]

 Communications/Transfers: Updated the transfer module and add support for retries that may need to happen on a transfer that is only 1 chunk large. May fix some failed cases of small transfers.
Communications/Transfers: Increased the tolerance for slow/trouble connections by lowering the minimum transfer rate to 2k/sec. Each item is just under 1k, so this should allow a rate of at least 2 items/second to continue to progress.
Communications/Transfers: For each transfer failure, the server will now shrink the send window by half, until the window becomes the minimum size of 1. Hopefully this will deal with people who have serious packet reordering problems.

(0035360)
SnootsDwagon (reporter)
2019-06-10 07:59
edited on: 2019-06-10 08:10

DATA ACCESS: Freaky, if as you say the asset sever isn't involved at all... that eliminates the middleman. This simplifies the process considerably, bringing it down to direct sim server to Viewer communications-- which should be faster than greased lightning.

LINDEN LAB. We don't have to get LL to cooperate with us. Right now we can be guaranteed LL is very much aware of Opensim and now considers us (if they have any sense) to be serious competition. Opensim warrants its own viewer... but there is MUCH EASIER way to go about this: logic forks. "If we're on Second Life do it this way... if we're on Opensim do it that way." This widely-used coding method is applied world-wide in programs operated in differing environments... and takes Linden Lab Corporate whims out of the equation.

We no longer have to hang on to the LL apron strings. We can forge our own path-- within the very same Firestorm viewer. Instead of doing it the Linden Lab way... we branch and do it the RIGHT way. No need to ask LL's permission or get them to agree at all. We just do it. ; )

TEST RESULTS: Be assured that the test run more than a year ago on Inworldz and Second Life both, is IDENTICAL to the test I ran yesterday... even using the same device. Rez an object filled with contents (in this case a music player), click the content tab, click the stopwatch. Not much room there for deviation of protocol. The testing parameters were exactly the same... as were the identical tests run on Second Life. In both tests, the Inworldz Viewer and Second Life Viewer out-performed Firestorm in fetching object Contents (by a VERY significant amount). This is empirical, repeatable research and resultant data. Anyone can repeat this experiment based on the parameters I've provided.

THE NET. The one thing we have little control over is the Net. That problem is somewhat solved by basic coding practice: if we hit an often-recurring and identifiable snag, code in routines to bypass and correct that snag. I do that all the time in basic in-world scripting... and have done so all my life in RL coding. The idea of re-sending packets is just such a concept: the first one didn't go through, so send it again. (I know this is basic "Coding 101" information... but points out there is always a way to fix something that's not working.)

All of these concepts combined (especially creating Opensim-specific code within Firestorm) could increase performance significantly-- which is the goal.

(0035361)
aiaustin (developer)
2019-06-10 08:05

@SnootsDwagon, there are already SecondLife and OpenSim code branches in the Firestorm viewer.
(0035362)
SnootsDwagon (reporter)
2019-06-10 08:07
edited on: 2019-06-10 08:08

Woot! Glad to hear it aiaustin. Then Firestorm is already on the right path. That is encouraging news.

Sometime today I will try to find time to repeat this viewer vs viewer test on SL, and will post the results for examination.

(0035364)
tampa (reporter)
2019-06-10 09:50

@SnootsDwagon While FS has an OpenSim version, said branch is not exactly to be considered an OpenSim-Viewer in the native sense. It's just a compatibility patch to retain or change minor code. What you are suggesting really requires fundamental changes on both ends.

The problem with that, such a viewer project would need to be maintained actively and for the foreseeable future, otherwise breaking compat with all the other viewers could lead to a situation where OpenSim as such has no supported viewer and becomes inaccessible.

You are also just re-iterating what has been said for years, what each years OpenSim conference keeps repeating and what many commercial forks have done in the past, this is no new information for us and the tests you conduct are already well documented or easily accessible through the viewer source.

If you truly want to help, I suggest try helping Dayturn as it seems to yet be the most promising candidate for an OpenSim-only viewer that could leverage some protocol changes.

I would also say that running parallel protocols for say "OpenSim-compat" and "OpenSim-direct" in terms of the viewers could potentially work. Say fallback to older standard protocol if viewer does not support newer caps. Still would mean major rewrite, a whole handling system for that and so on and so forth. Overall a massive undertaking the project currently has no real manpower for. Hence why that is generally something that was done by commercial forks like Inworldz and the like.
(0035365)
FreakyTech (reporter)
2019-06-10 09:54
edited on: 2019-06-10 10:03

@SnootsDwagon
How many items does your object have? Because my test object has 600 items in its contents and results into ~30s.

Using any other object is just comparing apples and peaches.

For comparision the numbers of contained items must be similar.

About your "We just do it". Find programmers that do it. That is the actual tricky part to get. And have the user base accept that newest SL features are no more to get into.

About InWorldz code change:
The first thing I did was experimentally using the way inworldz changed the code and that code triggered fast on packet reordering that the sliding window to become 1. Effectively disabling it. Afterwards, I wanted to know if there has been some kind of specific change in InWorldz viewer. However, the available source repositories related to inworldz on bitbucket.org do not hold any relevant code change to mitigate that.

The problem is not that linden lab is providing that change. But as long as you do not find volunteers to do full scale viewer work on all parts regarding the viewer, it simply stays wishful thinking to have an independent opensim viewer.

The FirestormOS branch is a mere spin-off branch to have certain OpenSim specifics inside.

As long as there is no opensim devs mandated reference viewer I do not see any such effort working out.

-----
Now a specific criticism on those inworldz changes:

That specific inworldz code missed the fact that the "reliable" packet flow already has acknowledges and therefore the viewer does not expect such upper layer packets to be resent independently.
A correct change would have been to tap into the acknowledge path of the "reliable" packet flow. Which is btw. also meaning the actual design weirdness it holds. And that already includes remarks about the illfated confirm mechanism the transfer protocol has for the inventory data.

-----
NB: I have no actual problem with changing such protocols but that means declaring all LL upstream dependent viewers to be incompatible and not to be used anymore.

And adding more and more protocols just contributes to that convoluted code base that the viewer is itself. All LL derived viewers btw.

-------
https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing [^]

(0035366)
UbitUmarov (administrator)
2019-06-10 10:15

if memory serves me right, main reason for viewers to start having a opensim specific binary had to do with mesh processing libraries. LL uses commercial havoc tools that cannot be used for opensim and is replaced by hacd and respective glue.
(0035367)
SnootsDwagon (reporter)
2019-06-10 10:17
edited on: 2019-06-10 10:23

Freaky, I started out a prior post as follows:

"I ran a test on OSgrid tonight. Here are the results:
Object containing 689 Contents
Time to load on multiple tests: between 26 and 33 seconds"

So our results are similar.

Regarding Inworldz code archives: they lost 27 OARs in the shutdown chaos. No one has any idea how they were lost but some were major OARs. So I wouldn't be surprised if the public release of Inworldz code is not complete, or did not contain the information specific to this issue. All I can say is that when I ran this same test on Inworldz, using the Inworldz viewer, the results were 6 to 8 seconds, consistently. I don't know how they did it. I don't have the information to debate it. I'm just a user reporting the facts.

The best position that I can speak from is not as an Opensim / Viewer dev (which I am not) but as a retired RL coder who has worked with more database access / security routines / network interfacing than one can shake a stick at. I know how quickly databases can be accessed, how quickly data can be transferred, and I know that accessing 600 pieces of data in 30 seconds is astoundingly slow.

This is verified by a developer /grid owner (whose name I won't drop here), who told me that such database access is measured in milliseconds, not seconds, and that with the speeds of today's equipment, by sheer logic one should be able to access 500 items in less than one second. This agrees with my own knowledge of database access.

I can also say with certainty, that if Amazon ran like Opensim and Second Life, they'd go out of business. That's not an insult by any means-- that is a realistic comparison, apples to apples.

So I understand what everyone is saying. If as Tampa claims this problem has been known and recognized for years... all that tells me is that it should have been fixed for years. Basic access to data is the foundation of any program. The 3D don't work if the 2D don't work. The 2D don't work if the data access don't work. Data access fails to work because of two reasons: either Network is down or malfunctioning... or proper data access protocols are not being followed.

That's the statement of a professional programmer, not an Opensim / Viewer dev. I do believe that Opensim and Firestorm both must by necessity follow programming protocols that have been designed and used for decades.

So... when I see a system taking 30+ seconds to access a mere 600 data items, when I see teleports failing because system-to-system handshaking and verification is not being performed properly (I can see it right there on my screen, folks), when I see cache not being used and when I see obvious logic flaws, even though I don't know specifically how Opensim and Firestorm code is written, I believe myself qualified to say: this can work lots better if someone has the time and desire to tackle it.

However, I'm not the one anyone needs to be convincing of anything here. My opinions in this matter are irrelevant, truly. I'm merely the guy crying out that the Emperor could be better-dressed. ;D

What is relevant is the observation, dedication and available time of the devs. This isn't a "convince Snoots of stuff" forum. It's a Mantis in which the Devs try to figure out if things could be done better and faster. As I was taught in business management classes in college, the key is to come up with solutions, not excuses. So that's how I view things: It isn't working properly... so what can we do to make it better?

I'm not the one to convince. I'm just helping to follow up on an issue that was reported in October 2018 (and honestly, long before that). I'm rooting for and willing to assist anyone that wants to tackle this bear and train it. : )

Current Second Life tests hopefully coming up today, if my time permits.

(0035368)
FreakyTech (reporter)
2019-06-10 11:30

Mine was fact stating as well. When I saw the hard-coded ack throttling in viewer I straight thought "WTF" for a good reason. Because without that part being changed there is simply no room to get that stuff much faster with sliding window logic.

I calculated from observed network trace data that I should be reaching something like 16s instead of 30s when that viewer induced 30ms delay per packet alone is gone. Essentially shaving of 30ms of the 70ms rtt observed in my case.

Getting a hard fork initiated (i.e. real opensim viewer) is not easy either. Try to find volunteers for that that are about to tackle besides coding topics also users crying for SL features. And as most devs doing viewer and/or simulator outside of LL realm are free-time volunteers it will be hard to get the needed commitment that is required to tackle such a hard fork.

E.g. if Firestorm devs manage to give LL a little push about improving that it will be just easy to get that into use later on. But a hard fork is a different deal to get sufficient people onto that project. Tackling differing graphics cards and systems is way harder than it sounds. Give yourself some time and research issues that well-distributed games had with graphics cards and their drivers.

About users requesting features:

see Display names => https://www.hypergridbusiness.com/2019/06/display-names-come-to-opensim/ [^]
see EEP => http://opensimulator.org/wiki/Chat_log_from_the_meeting_on_2019-06-04 [^]


If you want to get an opensim specific viewer into existence, support those people and put some effort in getting the required people at helm to keep it running.
(0035369)
SnootsDwagon (reporter)
2019-06-10 11:32
edited on: 2019-06-10 11:48

I just completed testing on Second Life. Entire test took just a tad over an hour. I tested the latest Firestorm in comparison with the latest Second Life Viewer. Test verifies that Firestorm has improved over the past few months. Whereas prior the SL Viewer performed significantly better than Firestorm in displaying object Contents, the two are now neck-to-neck. Results:

TESTS CONDUCTED IN HIGH-SKY RAGLAN SHIRE SILICA CUBE PARCEL

FIRESTORM:

In transferring inventory items (sound clips) to a basic cube and vice versa, received two error messages:

"Inventory creation on in-world object failed" when transferring from inventory to box.

"Cannot create requested inventory" when transferring Contents from a box to Inventory.

This seems to result from trying to transfer "too many" inventory items at a time. This problem vanished when inventory transfer was kept to around 30 items at a time (time consuming, but more consistently accurate).

* The box often failed to display all contents, requiring 2 or 3 efforts to get contents to display completely.

* It took an average of 25 to 32 seconds to ADD a single inventory item to Object Contents when contents got past 200 or so items.

TIME TO DISPLAY ALREADY LOADED CONTENTS IN EDIT MODE, 661 items:
Five tests, conducted one after another:
34 seconds
32 seconds
34 seconds
33 seconds
32 seconds

It is evident as Ubit points out, there is no Content caching taking place.

End of Firestorm Test on Second Life

SECOND LIFE VIEWER:
Five tests conducted on the same box, one after another:
34 seconds
34 seconds
34 seconds
33 seconds
35 seconds

As evident, the times compared to Firestorm are almost identical. Firestorm was occasionally a second or two faster, but SL Viewer more consistent in results. Neither one seemed to have advantage or disadvantage over the other.

Second Life Viewer exhibited same error messages as Firestorm, namely "Inventory creation on in-world object failed".

The good news: Firestorm now performs as well as the SL Viewer in handling object contents. This is a major improvement over when this Mantis was first filed.

Not so good news: neither viewer is handling data with the speed that would be expected, nor comes close to the 6 to 8 second speeds achieved on Inworldz using the Inworldz Viewer. Pointing this out because someone had to "set the bar", and Inworldz did so-- providing the target to aim for.

As a user, one nagging point continues to bother me: this is straight-forward data access, directly from the region server. Considering today's high-speed hard drives, RAM, busses and CPU engines... such a transfer should (as the tech I mentioned before verified) take around a second. We can add a second or two for "Virtual World Environment" and of course Net time to get from the Server to the Viewer.

However in my case, Server to Viewer has a ping time of about 4ms. That pretty much eliminates Net lag. That leaves the elusive "virtual world environment" as a variable, but it reasonable to assume that wouldn't seriously impact a less-than-one-second database access directly from the sim server.

Just observations, and the resultant tests I promised, for whatever value it may offer. My observations and opinion in this is based on my RL secular experience, and may not necessarily translate well to coding a virtual world environment-- which is admittedly an extremely complex coding project. : )

I still hold there is room for improvement. That said, performance is far better than when this Mantis was first reported. If someone could somehow knock this down to 6-8 seconds... or install an effective cache system, I believe we could call this Mantis item "fixed". As it stands, I consider the efforts and successes thus far in this area to be significant. Going from 3+ minutes (to infinity) down to a consistent 25 to 35 seconds is a major improvement.

(0035370)
beqjanus (reporter)
2019-06-10 12:21

As I stated in my very first post, the majority of the transfer time is composed of the latency fuelled by the nature of the protocol. We do indeed have an OS fork and it is brand new, having been (very reluctantly) created in recognition of the fact that the evolution of the lab protocols such as EEP is making it extremely hard to keep two divergent sets of technology in a single code base.

the reluctance in doing the split was centred around the very limited availability of viewer developers. I have agreed to help on the OS branch, maintaining and fixing where I can, but I am not anticipating any significant new work.

I also need to ensure that firestorm remains viable for the widest range of OpenSim distribution possible and I cannot commit to supporting an arbitrary invention without some form of canonical reference, doing so would set a precedent for accepting proposed solutions from any direction. The best I can do right now is to take up the cause with the lab and suggest work to improve the only true reference we do have. They may have no interest, but the time is actually in our favour as they are showing increased interest in improving the reliability of data transfer in accordance with their desire to move the entire platform into the cloud.
(0035371)
aiaustin (developer)
2019-06-10 12:29

Thanks for your work on this @beqjanus. I hope that the fork though does not lead to a potential future problem as these things do depend on willing and able individuals like yourself (and Cinder Roxley and others previously). And we are grateful to you for that work.

I have always been a strong advocate of OpenSim staying in step with Second Life where feasible for the reason that we are more likely to maintain a well supported viewer used by a far larger audience than OpenSim alone. But the occasional special branch of code makes sense so long as it can be compartmentalised and the main body of the code kept in synch with the main developments.

But its great to have a contact in the viewer development team able to think about these things. I can guess you will get the usual flack when some "obvious easy change" feature is requested, but hope you see positive feedback as well. Wishful thinking in these days of social media eh :-)
(0035372)
tampa (reporter)
2019-06-10 13:53

It all seems simpler than it really is at the end of the day, probably why I don't bother with doing a lot of things. It helps to have some backing and people standing behind you pushing on, though many overdo that.

Anyways. I was trying to replicate the loading tests to see what speeds I would get on a bare install with assets in db vs. ZetaWorlds with assets as php-fpm served files behind a cdn. The results were not that conclusive though, mainly due to network routes changing, but nonetheless after each new load the cdn cached the data and delivery happened nearly twice as fast. 200 items in 0000002:0000008 seconds or less on the first load, practically instantly the second time. Clearing viewer cache manually and switching to a different simulator/binary for clear assetcache.

I can still see some data being sent however, which may be due to the aggressive culling in the cdn, I have fear otherwise the assets get "lazy" in updating. I can also still see slow responses, though I suppose that is normal given the current load on the disks(ssd is the way to go for assets nowadays).

As I said, I can't quite get a stale environment to test against, but it still confirms to me that outside of protocol changes, there are some things that can be done to improve performance, but that all to me constitutes nothing more than a workaround and not a solution. Especially given the nature of the content nowadays.

Oh that is something I should share. In the month that bento was rolled out I saw the biggest increase in the asset server data size than I have all along. The data I have from 2016 up to now suggests an increase 20% above normal. There is a certain urgency in this discussion given that content is increasing in complexity and size. In order to keep pace with ever increasing fidelity an adjustment to how we handle data is bound to be needed. I suppose that is why LL decided to start from scratch and deal with data in smaller pools or instances rather than one giant warehouse full.

While I am not sure if that is the way to go, figuring out a way to potentially store data in different ways to not only handle ever growing storage demand, but also provider faster access. I do remember toying around with the idea of supplying assets through redis, but despite that working, it never really went anywhere and I can't remember where that code is. Another idea was to treat the viewer connection as closed network and capsule it off, a sorta vpn, unfortunately that would open it up for manipulation from the grid side creating a security risk if the grid side is not properly secured. Also breaks like half the network adapters out there.

@SnootsDwagon If you want you can repeat your tests on ZetaWorlds, I set the sandbox to the cdn, should give you less variables compared to osgrid.

@aiaustin It would probably be a good idea for someone to update the wiki with more information regarding the used protocols, where exactly the downfalls lie and so on. Being able to point to such a resource usually helps get people to move and implement an OpenSim change into viewers. If anything it could provide a spark to get something done.
(0035373)
FreakyTech (reporter)
2019-06-10 14:21

@Tampa Sorry you confuse download of visible prims with prim contents loading

prim contents loading just downloads the inventory without assets and you tested a viewer login to a region.

Different test case.

UDP does never run through your CDN setup.
(0035374)
tampa (reporter)
2019-06-10 14:42

Prim contents, in my case notecards and blank textures. Didn't want to mix it further. So no, not actually confused here.
(0035377)
FreakyTech (reporter)
2019-06-10 16:04

Just it is not about opening notecards nor blank textures. It is simply about the time it takes to show the actual list of prim contents.
No assets involved in that case.

Prim contents list is solely loaded via UDP means. And that is the protocol specification.
(0035378)
SnootsDwagon (reporter)
2019-06-10 21:53
edited on: 2019-06-10 21:56

Follow-up: I had been hesitant to quote by name the tech that I spoke to some time ago because I couldn't remember the exact conversation. But then I re-read this Mantis and well... there it is. So just to bring back to memory, going to quote from Balpien Hammerer again (owner of Discovery Grid), who is also the person I worked with testing the initial texture DDoS snafu on SL in 2007... and the guy who figured out exactly what was going on 3 years before LL coders "discovered" the problem themselves. Here's Balpien's quote repeated (there is more in the OP):


"I can do a MySQL query of 50,000 prims and have the query done in 1/10th of a second. There are two major bottlenecks. One is that OpenSIM's Robust's httpserver is anemic at best. It's a long known problem with OpenSim. So, if the asset fetches are coming form Robust it will be slow. InWorldz fixed a lot of that."

Unfortunately not much there as far as inside tech info... except I can say that Balpien was working with Inworldz when all this was going on. He's sharp. If he says it can be fixed, I would think he would be the person to question to see how it might be done. If this is a problem with Robust (and I could easily believe that is the case), the solution might be a totally different direction than being currently considered.

You're welcome Balpien, for me volunteering you. Don't shoot me. ;D

(0035379)
tampa (reporter)
2019-06-10 23:26
edited on: 2019-06-11 00:04

@SnootsDwagon You can always throw away what is currently there and start from scratch trying to implement a working solution around the protocol, but there is only so much that can be done in the constraints set by the protocol. Knowing the issue does not equate knowing a solution unfortunately, otherwise we would have one by now seeing as this issue has been known for years now. I even explored some ways to deal with it, see below, but ultimately you can't trick all the involved systems and one will complain and crash.

(0035380)
FreakyTech (reporter)
2019-06-10 23:46

Since this is now get derailed, I am discontinuing providing any further insights.

As it is not about anything that is dealt with caps as of now but sits straight in UDP data path and does not even access any asset. There is simply no CDN support for retrieving the inventory list on prim contents.
(0035382)
SnootsDwagon (reporter)
2019-06-11 05:55
edited on: 2019-06-11 06:13

Tampa and Freaky, I readily admit I don't understand the insides of the system. I don't know what you mean Freaky about this "getting derailed" as we are still discussing the OP topic. I'm just throwing ideas out there to see if any of them can be utilized.

Tampa, from what I understand, you're saying you folks are aware of the problem and have been working with it for years, but you can't redesign the engine-- which I understand. From what I gather it's like trying to work with Windows; there's only so much we can do and after that, it's the limitations of Windows itself that get in the way.

So am I correct in assuming that Robust is an engine we can't alter, and either have to work within its constraints or re-write that system entirely? That would certainly be a job of epic proportions and understandably nigh-impossible.

However I'm not understanding something: we are told above that Object Contents are not fetched from the Asset server, but from the sim server itself. Upon thinking about it I think that while that statement might be true technically, it's not true in concept.

I believe it is quite possible the Contents are being accessed from the Sim Server, yes. But the Sim server has to get those contents from the Asset server. so the first time that object is rezzed, the sim server has to do that Asset fetch. And if the asset server is running like a dog with two legs, that is going to be slow. That could explain a slow initial loading of the contents list... but doesn't explain it satisfactorily.

That doesn't explain why subsequent loads of contents take just as long-- because if that data is coming from the sim server (and thus bypassing Robust entirely from that point on)... the contents should be cached with the sim server and immediately available. BAM! Almost instantaneous display.

But if as Ubit says, the sim server is obtaining that informtion from the Asset server via Robust, then sending that info to the Viewer-- and then is not retaining that information-- then the next time the viewer asks to see those contents the sim server has to go right back to the Asset server and get that information again. Am I correct in that?

So in effect, while technically the Viewer is getting the information from the sim server, the reality is that information is coming from the Asset server and then handed from the sim server to the Viewer-- and thus we hit the REAL problems in this issue:

1) Robust is slow and not accessing assets quickly
2) The sim server is not caching assets as it should

This seems to be the heart of the issue. If we cannot alter the speed at which the assets are retrieved (for some reason Inworldz could but Opensim can't. I'll just take that as supposed reality)... then the solution would lie in the area of caching that information so the server doesn't have to do repeat asset server access every time we edit the object.

This becomes VERY important when we view this on a grid-wide basis. If the asset server is being accessed by the sim server every time an object is edited, and if that happens for every user on the grid accessing an object throughout the day-- that means the asset servers are being deluged with repeat requests for information the sim server has already accessed and should have cached (but didn't). This then, is exactly the same logic problem Balpien and I discovered on SL some 12 years ago-- fetching the same textures over and over and over again. Only in this case it is object Contents that are being re-fetched every time the builder accesses those contents. And that is what is slowing that process down to a snail crawl.

The evident solution: build a caching system for the region server so that it doesn't have to access the asset server every time the Contents are accessed. If a change is made to those contents, update both the cache and the information in the Asset Server so that both are kept up to date. A small coding job? No. A necessary job? Yes, if we ever want Opensim to be scalable and grow.

Again, I'm speaking from an "outside view" here and in plain English rather than technoese. Is there anything in what I've outlined above that is incorrect, unworkable or impossible? Or (based on all the information we've read above) have I pretty much hit the nail on the head?

/me puts away my sonic screwdriver

(0035383)
FreakyTech (reporter)
2019-06-11 06:16

The derailment came from the now edited post #35379.

Contents are only accessed on assets when opening an item.

However, this bug issue mentioned the loading of item list. And the item list is not stored on the asset server.

You still assume that loading the item list involves loading assets. Assets are not involved for displaying the item list.

Assets are cached on opensim region server already. Check the folder assetcache.

Item list in prim and assets itself are separate.

The item list just holds asset ids for referencing an asset but not the actual asset.
(0035384)
SnootsDwagon (reporter)
2019-06-11 06:32
edited on: 2019-06-11 14:24

Let me ask a question Freaky:

When a builder pulls an item from INVENTORY and edits the item, accessing the Contents tab... are you saying that item is stored in the sim server cache and that the asset server is not accessed? I would tend to believe the asset server is queried for all the information on that newly-rezzed item.

Now, it would make sense that once that item is rezzed the information about that item is then stored in the sim-server cache. That would be ideal. Then when making further inquiries about that item, the asset server would not be involved.

The question however: is that what is happening? When the user turns his/her attention away from that object, then later returns and edits the Contents... is the information that shows up in those contents coming from the sim server cache... or is it coming from the Asset server all over again?

Ubit states absolutely the sim server is not caching the object contents. Experimental data points to his claim as being likely true. It takes just as much time to do subsequent Contents access on following access as it does on initial access-- indicating the list of Contents is coming from the same source (the asset server), and that the sim server is querying the asset server all over again.

So which is true? On subsequent access of Object Contents-- is that list coming from sim server cache, or is it coming from the sim server re-accessing the asset server and pulling in the same data repeatedly? The time involved in displaying that data would indicate the later.

This answer to that question is something we need established to work on this further. Where exactly is the data coming from (initially) when the user performs Object Contents access? If it is coming directly from the sim server cache with absolutely no Asset server access, then I would have to say that is the slowest cache on the face of this planet and needs re-written.

What we're seeing here is two different claims: Ubit states the items are not being cached. Freaky is claiming they are being cached and the asset server not being further accessed. Which claim is true? Someone may wish to check the code on "subsequent object content access" and see where the data comes from. Does it come strictly from the sim server, or is a query made to the sim server, which then pulls the data (again) from the asset server, and then sends that data to the viewer?

Experimental data indicates the later (very slow) concept is the reality.

If as Ubit claims, that cache is not in place, the logical solution would be to put a cache in place so that this problem is corrected. If we can't fix the speed at which assets are accessed... then make best use of that data once it is accessed so that the asset server is not repeatedly queried for identical data.

(0035385)
UbitUmarov (administrator)
2019-06-11 08:07
edited on: 2019-06-11 08:50

Freaky is right, this issue did derail a lot.
The issue is well understood by those who need to try to improve it, and was explained possible in excessive detail already.
- We were very clear telling the problem is on a very specific viewer<->server protocol
- We were very clear telling that opensim server has the improvements that are possible at this point.
- We had viewer dev that kindly is looking to the issue, viewer side.

So thing there is nothing more to say here, until there are news from devs side
I just do not close this now, because of that
If you have other issues, please file respective mantis

(0035389)
SnootsDwagon (reporter)
2019-06-11 14:34
edited on: 2019-06-11 15:17

I understand what you're saying Ubit. However, "opensim server has the improvements that are possible at this point" is the debatable point.

The question is still open: does the Opensim software cache object contents or does it not? If it does not... then has everything has been done that can be done to make this faster?

Regarding the "one Lego at a time" illustration. I admit this discussion got to a point that the technotalk lost me, but is there a way to fix that "one at a time" situation? Or is this something already agreed to check in to?

Just curious to have those two questions answered so us regular users can understand. I did file the original Mantis and have run some definitive tests... so it's understandable I'd like to understand what's happening as well. ;)


- Issue History
Date Modified Username Field Change
2018-10-06 16:06 SnootsDwagon New Issue
2018-10-06 19:27 UbitUmarov Note Added: 0033131
2018-10-06 19:30 BillBlight Note Added: 0033132
2018-10-06 19:32 UbitUmarov Relationship added related to 0008202
2018-10-06 19:34 UbitUmarov Relationship added related to 0007904
2018-10-06 20:56 SnootsDwagon Note Added: 0033133
2018-10-06 20:57 SnootsDwagon Note Edited: 0033133 View Revisions
2018-10-06 21:03 SnootsDwagon Note Edited: 0033133 View Revisions
2018-10-06 21:04 SnootsDwagon Note Edited: 0033133 View Revisions
2018-10-06 21:09 SnootsDwagon Note Edited: 0033133 View Revisions
2018-10-06 21:17 SnootsDwagon Note Edited: 0033133 View Revisions
2018-10-06 22:24 SnootsDwagon Note Added: 0033134
2018-10-07 09:44 BillBlight Note Added: 0033141
2018-10-07 10:24 SnootsDwagon Note Added: 0033143
2018-10-07 10:32 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:34 SnootsDwagon Note Edited: 0033134 View Revisions
2018-10-07 10:36 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:36 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:36 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:48 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:49 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-07 10:57 SnootsDwagon Note Edited: 0033143 View Revisions
2018-10-08 09:46 UbitUmarov Note Added: 0033150
2018-10-08 13:00 SnootsDwagon Note Added: 0033151
2018-10-08 13:01 SnootsDwagon Note Edited: 0033151 View Revisions
2018-10-08 13:03 SnootsDwagon Note Edited: 0033151 View Revisions
2018-10-08 13:04 SnootsDwagon Note Edited: 0033151 View Revisions
2018-10-08 13:12 BillBlight Note Added: 0033152
2018-10-08 13:12 BillBlight Note Edited: 0033152 View Revisions
2018-10-08 15:15 SnootsDwagon Note Edited: 0033151 View Revisions
2018-10-09 01:45 aiaustin Note Added: 0033153
2019-06-07 16:25 beqjanus Note Added: 0035324
2019-06-07 18:26 SnootsDwagon Note Added: 0035325
2019-06-07 18:33 SnootsDwagon Note Edited: 0035325 View Revisions
2019-06-07 18:33 unregi Note Added: 0035326
2019-06-07 18:41 SnootsDwagon Note Edited: 0035325 View Revisions
2019-06-07 18:42 SnootsDwagon Note Edited: 0035325 View Revisions
2019-06-08 05:20 beqjanus Note Added: 0035330
2019-06-08 05:58 SnootsDwagon Note Added: 0035331
2019-06-08 06:09 SnootsDwagon Note Edited: 0035331 View Revisions
2019-06-08 06:18 SnootsDwagon Note Added: 0035332
2019-06-08 06:21 SnootsDwagon Note Edited: 0035332 View Revisions
2019-06-08 06:26 aiaustin Note Added: 0035333
2019-06-08 06:29 aiaustin Note Edited: 0035333 View Revisions
2019-06-08 06:34 aiaustin Note Edited: 0035333 View Revisions
2019-06-08 06:38 aiaustin Note Edited: 0035333 View Revisions
2019-06-08 06:40 aiaustin Note Edited: 0035333 View Revisions
2019-06-08 06:50 FreakyTech Note Added: 0035334
2019-06-08 06:51 FreakyTech Note Edited: 0035334 View Revisions
2019-06-08 06:51 FreakyTech Note Edited: 0035334 View Revisions
2019-06-08 10:08 UbitUmarov Note Added: 0035335
2019-06-08 10:09 UbitUmarov Note Edited: 0035335 View Revisions
2019-06-08 10:31 FreakyTech Note Added: 0035336
2019-06-08 12:32 UbitUmarov Note Added: 0035337
2019-06-08 13:52 beqjanus Note Added: 0035338
2019-06-08 14:11 beqjanus Note Edited: 0035338 View Revisions
2019-06-08 14:21 UbitUmarov Note Added: 0035339
2019-06-08 15:51 FreakyTech Note Added: 0035340
2019-06-08 16:04 UbitUmarov Note Added: 0035341
2019-06-08 16:06 UbitUmarov Note Added: 0035342
2019-06-08 16:15 UbitUmarov Note Edited: 0035341 View Revisions
2019-06-08 16:27 UbitUmarov Note Added: 0035343
2019-06-08 16:52 SnootsDwagon Note Added: 0035344
2019-06-08 16:57 SnootsDwagon Note Added: 0035345
2019-06-08 16:58 SnootsDwagon Note Edited: 0035345 View Revisions
2019-06-08 16:59 SnootsDwagon Note Edited: 0035345 View Revisions
2019-06-08 16:59 SnootsDwagon Note Edited: 0035345 View Revisions
2019-06-08 17:00 SnootsDwagon Note Edited: 0035345 View Revisions
2019-06-08 17:06 SnootsDwagon Note Added: 0035346
2019-06-08 17:07 SnootsDwagon Note Edited: 0035346 View Revisions
2019-06-08 17:14 UbitUmarov Note Added: 0035347
2019-06-08 17:26 UbitUmarov Note Added: 0035348
2019-06-09 04:37 FreakyTech Note Added: 0035351
2019-06-09 05:05 FreakyTech Note Edited: 0035351 View Revisions
2019-06-09 06:16 FreakyTech Note Edited: 0035351 View Revisions
2019-06-09 12:05 UbitUmarov Note Added: 0035356
2019-06-09 21:36 SnootsDwagon Note Added: 0035358
2019-06-09 21:39 SnootsDwagon Note Edited: 0035358 View Revisions
2019-06-10 03:21 FreakyTech Note Added: 0035359
2019-06-10 04:00 FreakyTech Note Edited: 0035359 View Revisions
2019-06-10 07:59 SnootsDwagon Note Added: 0035360
2019-06-10 08:00 SnootsDwagon Note Edited: 0035360 View Revisions
2019-06-10 08:00 SnootsDwagon Note Edited: 0035360 View Revisions
2019-06-10 08:03 SnootsDwagon Note Edited: 0035360 View Revisions
2019-06-10 08:05 aiaustin Note Added: 0035361
2019-06-10 08:06 SnootsDwagon Note Edited: 0035360 View Revisions
2019-06-10 08:07 SnootsDwagon Note Added: 0035362
2019-06-10 08:08 SnootsDwagon Note Edited: 0035362 View Revisions
2019-06-10 08:10 SnootsDwagon Note Edited: 0035360 View Revisions
2019-06-10 09:50 tampa Note Added: 0035364
2019-06-10 09:54 FreakyTech Note Added: 0035365
2019-06-10 09:59 FreakyTech Note Edited: 0035365 View Revisions
2019-06-10 10:03 FreakyTech Note Edited: 0035365 View Revisions
2019-06-10 10:15 UbitUmarov Note Added: 0035366
2019-06-10 10:17 SnootsDwagon Note Added: 0035367
2019-06-10 10:22 SnootsDwagon Note Edited: 0035367 View Revisions
2019-06-10 10:23 SnootsDwagon Note Edited: 0035367 View Revisions
2019-06-10 11:30 FreakyTech Note Added: 0035368
2019-06-10 11:32 SnootsDwagon Note Added: 0035369
2019-06-10 11:41 SnootsDwagon Note Edited: 0035369 View Revisions
2019-06-10 11:44 SnootsDwagon Note Edited: 0035369 View Revisions
2019-06-10 11:46 SnootsDwagon Note Edited: 0035369 View Revisions
2019-06-10 11:48 SnootsDwagon Note Edited: 0035369 View Revisions
2019-06-10 12:21 beqjanus Note Added: 0035370
2019-06-10 12:29 aiaustin Note Added: 0035371
2019-06-10 13:53 tampa Note Added: 0035372
2019-06-10 14:21 FreakyTech Note Added: 0035373
2019-06-10 14:42 tampa Note Added: 0035374
2019-06-10 16:04 FreakyTech Note Added: 0035377
2019-06-10 21:53 SnootsDwagon Note Added: 0035378
2019-06-10 21:54 SnootsDwagon Note Edited: 0035378 View Revisions
2019-06-10 21:55 SnootsDwagon Note Edited: 0035378 View Revisions
2019-06-10 21:56 SnootsDwagon Note Edited: 0035378 View Revisions
2019-06-10 23:26 tampa Note Added: 0035379
2019-06-10 23:46 FreakyTech Note Added: 0035380
2019-06-11 00:04 tampa Note Edited: 0035379 View Revisions
2019-06-11 05:55 SnootsDwagon Note Added: 0035382
2019-06-11 05:56 SnootsDwagon Note Edited: 0035382 View Revisions
2019-06-11 06:01 SnootsDwagon Note Edited: 0035382 View Revisions
2019-06-11 06:02 SnootsDwagon Note Edited: 0035382 View Revisions
2019-06-11 06:03 SnootsDwagon Note Edited: 0035382 View Revisions
2019-06-11 06:06 SnootsDwagon Note Edited: 0035382 View Revisions
2019-06-11 06:13 SnootsDwagon Note Edited: 0035382 View Revisions
2019-06-11 06:13 SnootsDwagon Note Edited: 0035382 View Revisions
2019-06-11 06:16 FreakyTech Note Added: 0035383
2019-06-11 06:32 SnootsDwagon Note Added: 0035384
2019-06-11 06:34 SnootsDwagon Note Edited: 0035384 View Revisions
2019-06-11 06:36 SnootsDwagon Note Edited: 0035384 View Revisions
2019-06-11 08:07 UbitUmarov Note Added: 0035385
2019-06-11 08:50 aiaustin Note Edited: 0035385 View Revisions
2019-06-11 14:22 SnootsDwagon Note Added: 0035388
2019-06-11 14:23 SnootsDwagon Note Edited: 0035384 View Revisions
2019-06-11 14:24 SnootsDwagon Note Edited: 0035384 View Revisions
2019-06-11 14:27 SnootsDwagon Note Deleted: 0035388
2019-06-11 14:34 SnootsDwagon Note Added: 0035389
2019-06-11 15:17 SnootsDwagon Note Edited: 0035389 View Revisions


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker