Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008516opensim[REGION] OpenSim Corepublic2019-04-09 03:312020-01-02 04:07
Reporteraiaustin 
Assigned ToUbitUmarov 
PrioritynormalSeveritymajorReproducibilitysometimes
StatusassignedResolutionopen 
PlatformPCOSWindowsOS Version10
Product Versionmaster (dev code) 
Target Versionmaster (dev code)Fixed in Version 
Summary0008516: Missing Objects and/or Terrain Parts on Teleport between Grids
DescriptionThis is a Mantis issue to act as a place to note observations as the recent code changes related to SupportViewerObjectsCache are tried out more.

With recent dev master code on teleport there can be loss of objects and/or terrain chunks. They cannot be made visible by clicking in the area where the lost objects or terrain is known to be (which can occur already with some objects in viewers). So we are not able to get them back without moving away and back or relogging.

Sending a "force update" from the Server console also has no effect.

The recent commits suggest turning down the viewer bandwidth if this sort of loss occurs. I turned the viewer bandwidth down to 750KBps and that did seem to improve things a lot... but I still do see some items go AWOL occasionally even at 750Kbps. I have yet to identify the specifics of such a situation as its not all the time now.

I have also seen a couple of viewer crashes on teleport even at 750KBps bandwidth but again cannot be specific yet.

I do note a blip in packet loss in the stats tool (ctrl+shift+1) when the issues arises... Packet loss is on zero but as the loss occurs there is a sudden spoke a few seconds after arriving on a region where object/terrain loss can be observed.
Steps To ReproduceTeleport back and forth between grids running latest dev master code and observe if any objects or terrain patches are missing. Try viewer bandwidths such as 1500KBps (default for Firestorm on my setup) and 750KBps.
Additional InformationUbitUmarov and myself have discussed this quite a bit and done some cross grid hypergrid testing with a range of viewer bandwidths rather than my usual (the default I think) 1500Kbps I use on Firestorm (on a 100Mbps link at home and gigabit networking at work).

This occurred with both with SupportViewerObjectsCache set to false and true (true now being the default in master code). Ubit indicated the mechanism changed even if the parameter was set to false.

Ubit indicates that we should be resilient to reasonable packet loss. I have seen moderate packet loss on Second Life and on earlier OSGrid releases without any problems of such object loss. But here the visual loss of objects does seem to be associated with an observation in the stats console of a sudden spoke in packet loss as it occurs.
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Script Engine
Environment.NET / Windows64
Mono VersionNone
Viewerdev master (2019-04-09)
Attached Filesjpg file icon 2019-04-09-Terrain-Loss-1.jpg [^] (89,528 bytes) 2019-04-09 08:10


jpg file icon 2019-04-09-Prim-Loss-1.jpg [^] (182,032 bytes) 2019-04-09 08:22

- Relationships
related to 0008515resolvedUbitUmarov Mesh disappears 
related to 0008528assignedUbitUmarov Random mesh pieces do not rez 

-  Notes
(0035073)
aiaustin (developer)
2019-04-09 08:07

The way the stats for ping loss show it is as if the cause of the packet loss is occurring just after the arrival of the avatar at the destination region. Not that packet loss that just happens to occur causes the loss if the objects or terrain.

Packet loss can be registering zero, and seems to stay at zero whenever a successful TP takes place and no object or terrain is lost. But I think it is the case that each time I see packet loss if I look round I find there are missing prims or missing chunks of the terrain that are around 16m X 16m.
(0035074)
aiaustin (developer)
2019-04-09 08:21
edited on: 2019-04-09 08:23

Now I am observing something I find weird... note I am TPing between AILand grid Ailand region (avatar logged in natively there) and Openvue grid Openvue region (across Hypergrid). These are the two main regions on each grid. I am on latest dev master on each opensim-0.9.0.1-962-g49fb9d6.zip (2019-04-09 12:05). I am going slow on TP rate, leaving 20 seconds between TPs.

Once I get the prims missing (in this case three ground level 10x10 prims in the cobblestone area on Openvue as shown in attached image 2019-04-09-Prim-Loss-1.jpg), I can TP back home and return to Openvue and the SAME prims are missing. I can even relog, do the same and they can be missing.

BUT, and this is the odd thing... but might hold a clue to where the prims are getting lost... if when on the foreign grid Openvue region I TP via the map to an adjacent region (Sandbox) and the n return via the map to Openvue all on the same grid, the prims show.

BUT, then I go home to AiLand and TP back to Openvue again (all leaving 20 seconds between jumps) and the SAME prims have gone again. Its as if a totally different mechanism is being used to refresh the contents for inter-grid to inter-region jumps. I told you it was weird.

(0035075)
UbitUmarov (administrator)
2019-04-09 08:45

been there via HG with 1500kbps fs and had no issues
(0035076)
aiaustin (developer)
2019-04-09 08:47
edited on: 2019-04-09 08:53

Good :-) I am watching the logs as you hop about Ubit.

All looks normal. This showed once...

16:53:03 - [LLUDPSERVER]: Received a resend of already processed packet 0000047, type RegionHandshakeReply from Ubit.Umarov @hg.osgrid.org

(0035077)
mewtwo0641 (reporter)
2019-04-09 22:03

I see this issue still on master with grid set up as not HG. Have to constantly issue 'force update' command in the console to get the missing pieces to show.
(0035078)
aiaustin (developer)
2019-04-10 02:04
edited on: 2019-04-10 02:13

I updated to opensim-0.9.0.1-966-gee989dd.zip (2019-04-09 23:27) for both grids and observe essentially the same issues. At 750Kbps it works most of the time, but at 1500Kbps it fails most of the time. And after I have prims and terrain missing even if I wait 20 seconds between a TP out and back the same bits are still missing.

As before a small packet loss just once was associated with the first appearance of the missing prims/terrain chunks and after that it sticks even with nno further reported pin loss. It looks like once lost its.

BUT.. and I still find this peculiar and perhaps a hint as to what might be happening... a TP to an adjacent region (Sandbox) on the second grid and back to the region on that grid with missing prims/terrain (Openvue) makes everything show fine, but TP back to home grid (AiLand) and back to Openvue and the same bits are missing again.

I have double checked, and "force update" has no effect on the missing prims or terrain chunks for me on Openvue/AiLand grid testing. I think MeMewToo0641 may have a separate issue as in all my testing I have never been able to make any prim or terrain chunk that goes missing reappear.

(0035079)
aiaustin (developer)
2019-04-10 02:16
edited on: 2019-04-10 02:19

Ohoh… now the missing chunks and prims are persisting across relogs. I logged out and back in and after all settled did a jump to Openvue grid and the same bits are missing there.

In two separate relogs, I logged in to AiLand and let everything settle down. then I did my first jump to Openvue grid.. and the parts missing on previous logins were still missing. Same parts.

(0035080)
aiaustin (developer)
2019-04-10 09:09
edited on: 2019-04-10 09:10

I still have the same prims missing on FIRST teleport from AiLand grid to Openvue grid after updating from yesterday's git master (966 - 2019-04-09 23:27) to today's (967 - 2019-04-10 13:01) and hence restarting the servers.

Is this possible? I thought the object cache got refreshed after server restart?

(0035081)
UbitUmarov (administrator)
2019-04-10 09:19
edited on: 2019-04-11 09:41

no direct relation to objects cache.
as I told several times, viewer just does not rez things it did received.
most likely because it did overload, leaving things in memory in broken state.
the effect ranges from totally missing, just invisible that show up on touch and even just a face or two visible.
What prims, depends just on order they did arrive to viewer and how the issue was triggered.

similar for terrain.

viewer bandwidth is not only about network, is also about viewer capability to process things.

(0035082)
aiaustin (developer)
2019-04-10 10:03

If it is in memory Ubit, why does it persist over relogs/viewer restarts, and even after server clean upgrades and restarts?

Also, how come the objects show when moving off to an adjacent region and back on the destination grid, yet the same missing parts are still missing after return to home grid and back. Is there a different mechanism for updates in these two cases?

Clearly something serious has gone wrong.. we have not seen issues like this before have we? Its a very different issue to the occasional non-rez of objects that show when the area they are in is clicked on.
(0035083)
BillBlight (developer)
2019-04-10 10:19

@aiaustin, the same could be said, for WHY am I and others on my grid, on osgrid, and on 2 test grids unable to replicate your issue ..

Stomping your foot and saying it's broken does not help the issue.

I can give you 4 cases where it is not broken.

Not trying to say it is not an issue somewhere but two people having an issue does not point to a systemic issue.
(0035084)
aiaustin (developer)
2019-04-10 11:03

Bill.. I am not “stomping my foot”. I am reporting a clear change in behaviour on my test grids. I am delighted that you don’t see any issue on your specific test environments. This Mantis Issue is here to collect information as and when or if anyone else starts to see the issues, so we can pin down the problem.
(0035085)
aiaustin (developer)
2019-04-13 01:00
edited on: 2019-04-13 01:01

Updated Openvue and AiLand grids to opensim-0.9.0.1-971-gcfd3923.zip (2019-04-11 23:19) and set SupportViewerObjectsCache = false explicitly in OpenSim.ini (default now being true from openSimDefaults.ini).

Testing over the last 2 days in two viewer environments one of which is on a fast link on the same subnet as the servers and the other is over a home ISP accessing the servers remotely. using Firestorm 6.0.2.56680 (Windows 10 Pro), default bandwidth of 1500Kbps.

I do not see any missing prims or terrain chunks on teleports between these grids with SupportViewerObjectsCache = false. I even tested TPs back and forth 5 or 6 times at under 5 seconds delay. All looks fine. No packet loss blip occurs when using the stats console.

I then restarted the two grids with SupportViewerObjectsCache = true on both and then see a few missing prims when doing TPs. The prims this time differed a little on each TP, but most were apparently sub-parts of a linked object.

I did these tests after seeing the commit opensim-0.9.0.1-968-gd977613.zip (2019-04-10 20:28) which may have fixed the problem with the issue occurring even when SupportViewerObjectsCache = false as reported earlier.

(0035086)
BillBlight (developer)
2019-04-13 06:25

Just for information purposes, FS default bandwidth is 500Kbps not 1500.
(0035092)
aiaustin (developer)
2019-04-13 09:57
edited on: 2019-04-13 10:08

Ah, that’s good to know Bill. I was just concerned to make sure we work well with defaults, as many users will never change such settings. I guess I set mine to “cable” speed a long time ago and did not realise that continued to be set.

(0035288)
aiaustin (developer)
2019-06-05 11:19

This issue now seems to be fixed.

Openvue and AiLand grids now use SupportViewerObjectsCache = true (the default as set in openSimDefaults.ini).

Issue is addressed on latest dev master - sometime between 24th March 2019 and 14th May 2019. So OSgrid version as of 14th May 2019 is good too.
(0035381)
aiaustin (developer)
2019-06-11 01:55
edited on: 2019-06-11 02:00

As at OSGrid 2019-05-14 I do see missing prims at initial login. Not TP or HG related. Simple cubes missing. Different ones in same linkset for example none after several logins. As before, right clicking or trying to operate on the affected object does not make the missing piece rez. A relog is needed.

(0035392)
UbitUmarov (administrator)
2019-06-11 22:31
edited on: 2019-06-12 00:32

let me remark that viewer capability to handle prims may depend on fps
try to keep the viewer maximized and selected during all login/tps

(0035517)
mewtwo0641 (reporter)
2019-07-31 01:45

Still see this issue on master and have had reports of it from people who use the grid. Setting SupportViewerObjectsCache = false doesn't seem to make a difference. I have suggested to the people reporting it to me to try the old zoom out and ESC zoom in trick; sometimes that works and sometimes it doesn't. Toggling Render > Volume in the Advanced menu (CTRL + ALT + SHIFT + 9) off and then back on again sometimes helps too, but typically I have to resort to sending the 'force update' command in the OpenSim console when they tell me that nothing is working on their end to fix it.
(0035595)
mike.dickson (reporter)
2019-08-20 06:46

We're also seeing this issue on fairly current master. Turning bandwidth down does appear to help but not eliminate the problem. It affects both mesh and regular prims.

I understand that people feel this might be a viewer capacity issue. However the issue DOESN'T happen on SL so I'm not convinced "bandwidth" is the fix. My viewer is rendering 100fps and with network lowered to 750 I still get dropouts sometime. Originally I thought kills were being sent but the prims are "there" (a physics shape often exists) but they aren't rendering.
(0035596)
mike.dickson (reporter)
2019-08-23 06:48

Interesting to note. If I enable the GetAssets cap handler this problem seems to essentially go away. Of course that means less data being shoved over UDP so that makes some sense. The ViewerAssets cap has been in place in the viewers for some time and UDP asset fetching is totally off in SL so its very possible the viewers don't expect to get this data over UDP and because of that stuff breaks.
(0035597)
UbitUmarov (administrator)
2019-08-23 12:06

Thanks for the information.
Meshes and Textures should already be out of udp, and those should be the bulk of assets data at arrival.
Never the less i did add the ViewerAssets cap, not only because it makes sense to send all assets via tcp(http) but also to prevent potencial future issues in viewers side code merging.
But it the problem is related to bandwidth. If got worse after a few changes i made improving lludp i had to reduce the effciency of udp packet usage (not using all MTU bytes) to mitigate it
Now im using a suicidal 3000Kbps (not recomended) and see no problems either on my test grid or osgrid. (in fact most regions i visit are on master with that cap active)
(0035598)
UbitUmarov (administrator)
2019-08-23 12:10
edited on: 2019-08-24 04:08

It may happen it is the extra latency of http that gives viewers a little more breathing time..

(0035599)
aiaustin (developer)
2019-08-23 13:21
edited on: 2019-08-23 13:25

On the assumption that users are directed to use Firestorm 6.0.2 can I check what we should set then? Do we just uncomment this OpenSimDefaults.ini line ...

    ; Cap_GetAsset = "localhost" DO not ucoment this line. Some popular viewers still dont do it right for opensim. Here to easy testing

Is the capability you mention for ViewerAssets just in your test code at present Ubit and not in dev master?

(0035601)
UbitUmarov (administrator)
2019-08-24 02:17

Yes think fs 6.0.2 has it broken
It only asks it on first region, then keeps using the same url
that of course may fail.
(0035602)
aiaustin (developer)
2019-08-24 04:08
edited on: 2019-08-24 08:10

Oh, that is quite serious. Does "localhost" map to whichever server the OpenSim.exe region(s) are running on, or does it resolve to ONE specific server on first use and then stick to that?

If its the latter, then that means it should really not be used except for grids that use a single server for everything.

I see it is indeed reported in the Firestorm JIRA and was corrected by Beq Janus back on 28 Feb 2019 in the Firestorm code... the problem being of course that this is after the current Firestorm 6.0.2 release for OpenSim.

https://jira.firestormviewer.org/browse/FIRE-23657 [^]

(0035603)
aiaustin (developer)
2019-08-24 08:06

Is the OpenSim server side Cap_GetAsset capability the very same as the “ViewerAssets” capability or are they different?
(0035604)
UbitUmarov (administrator)
2019-08-24 13:36

It is the same
(0035605)
Total Sorbet (reporter)
2019-08-24 13:43
edited on: 2019-08-24 13:52

On login to OSG i have 25% of prims missing.. If i then jump to an alternate region viewer will show nearly all prims (100% on a good day :p) I have tried various bandwidth settings and although i would say this definitely has an effect im unable to find a setting that fixes this.

My default login procedure recently is to login to somewhere i don't want to be then jump to somewhere I do! That way I get to see most prims :)

(0035606)
UbitUmarov (administrator)
2019-08-24 15:04

Total forgot to mention she uses Linux Firestorm
(0035607)
UbitUmarov (administrator)
2019-08-24 15:58
edited on: 2019-08-24 16:04

Think i still had a silly bug on object cache support
Can you pls test master?
Clear viewer cache!!
Issues related to that bug should happen on second relog after that (or returning to a region already in viewer cache)

(0035608)
UbitUmarov (administrator)
2019-08-24 16:37

About the ViewerAsset fs problem. That is still not that relevant, old LLUDP( and http getTexture nd getMesh*) methods are still there, and used by most viewers, some not having that cap.

i did add it, because it does make sense to use http on those, and since reference viewers code had lludp methods removed, less viewers code merging issues to worry about.
(0035609)
mike.dickson (reporter)
2019-08-24 17:34

Had a quick chat with Beq who confirmed there was a bug fixed around ViewerAssets and the behaviour (cap not getting refreshed on a transfer) in FS. It's in 6.2 but not yet released for OpenSim.
(0035610)
aiaustin (developer)
2019-08-25 09:56

I updated Openvue and AiLand grids to latest dev master, cleared my viewer(s) cache(s) and will now watch for any issue,
(0035612)
aiaustin (developer)
2019-08-27 03:53
edited on: 2019-08-27 03:57

Just to report back... As well as Openvue and AiLand grids, I updated my OSGrid addon regions to the latest dev master also for testing. To date after the update to the latest dev master at 2019-08-25 and revisiting already visited (cached) regions, moving between the three grids, and relogs I have not yet seen any missing prims or mesh (or indeed terrain holes). I will continue to monitor carefully and report if I do see anything after this update.

It might be useful to encourage the OSGrid folks to update their distribution for add on regions so we better test this. I know people can compile the latest dev master and add the bin folder from that over the OSGrid version to update (as I have done), but possibly not many will do that.

(0035614)
UbitUmarov (administrator)
2019-08-27 10:53

Yes, code is just in "burnin" on some plazas before Dan makes a release
Thanks again for your testing and feedback
(0035622)
Total Sorbet (reporter)
2019-08-28 08:38

I updated and happy to report no missing prims past 2 days :)
(0035625)
mike.dickson (reporter)
2019-08-31 05:04

Just a quick update. I've rolled out to a few troublesome regions a build based on master with the suggested changes. I can report that for the updated regions this issue seems mostly addressed. I did have an issue with one region with a large mesh build (linkset) and a number of adjacent regions. When camming well back from the object when I zoomed back to my current position the building I was standing on was invisible. Edit, touching, etc, didn't bring it back. I don't think this was a case of the server sending a kill because the object was still physically there. Maybe something with interest lists. But the phenomenon was the same as the missing prims before this fix which I thought was interesting.
(0035626)
UbitUmarov (administrator)
2019-08-31 06:37

Server side culling does not suport camera position, only avatar position.
SO it may cause what you report.
maybe you have it active? (don't remember def as i type)

camera position support would be insane heavy...
(0035631)
mike.dickson (reporter)
2019-09-01 05:43

ObjectsCullingByDistance is false in the code and no default is set in the config or defaults files. But yes its acting that way. It's especially prevalent with Mesh and in some cases you get the invisible polygon that happens when a shape isn't loaded. In those cases edit with cause it to pop in. In others it's just gone. and there's no shape there to edit. So yeah these might be viewer generated kills because something is outside DD. I'll do some more experimenting. Overall better but performance is slow and this camera issue is a real hit for perceived reliability.
(0035633)
UbitUmarov (administrator)
2019-09-01 06:58

Maybe that mesh just has bad LODs?
(0035656)
mewtwo0641 (reporter)
2019-09-02 18:52

Have been testing master for past week; the issue seems to be much improved although I still see it on occasion and have to issue the 'force update' command to fix it. I am still testing with SupportViewerObjectsCache = false though. Should I be testing with it set to true, or does it matter?
(0035658)
aiaustin (developer)
2019-09-03 02:29
edited on: 2019-09-03 02:30

As others have said, I think we are seeing a range of issues when objects don't show.

For the original problem reported in this mantis "force update" had no effect, and there were no outline (physics style wireframes), or ability to touch any area and make the object, prim, terrain piece or mesh appear. I have not see that problem since Ubit made the latest changes. Running with 1500kbps and SupportViewerObjectsCache = true.

(0035676)
mewtwo0641 (reporter)
2019-09-07 18:32
edited on: 2019-09-07 18:32

@aiaustin - That is possible, I have been seeing a high number of log messages that look like this: "[LLUDPSERVER]: Received a resend of already processed packet (Number), type (Type) from (Avatar Name)" upon login of an account. The types vary from General, RegionHandleRequest, GenericMessage, AgentHeightWidth, AgentUpdate, AgentAnimation, EconomyDataRequest, and probably more. I've always seen these messages from time to time but they have never been this frequent or numerous as they have been in the past couple weeks of testing master and the high frequency of them seems to coincide with seeing missing objects upon login or it could just be anecdotal. It may also be entirely unrelated to this issue and/or the issue I described in my previous post. I am just unsure at this point in time.

(0035922)
gimisa (reporter)
2019-12-03 10:33

I notice the same difficulty in getting prim or terrain to show up completely.

Testing 091 official release 091 in gimisaOS grid under linux ubuntu 16.04 with mono version 6.4.0.198 .

I use viewer singularity alpha issue 6994 dating 2018-04-9 . I have been using this viewer since release. I also tried cool viewer, 1.26.4 this week release reading above that the issue was possibly with firestorm.

It is interesting to note that I have one computer that I could not update cause too old. That computer runs ubuntu 12.04 with mono version 3.2.8 and gimisa3 region simulator version 0901 official release. The rendering there is without issue.

Let me know if I can provide more data that could be usefull to solve that issue.
(0035925)
Ferd Frederix (reporter)
2019-12-03 11:06

What advice is there for this now, under 0.9.1 or dev master?
I still have run at 500 mb to see things reliably in Firestorm viewer, and even then there are times it does not show half the land and a lot of mesh is missing. TP away and come back and there will be more. 0% packet loss at any speed.

I currently have set the following in Opensim.exe.config:

 <useLegacyJit enabled="1" />

And in Opensim.ini have
[ClientStack.LindenUDP]
SupportViewerObjectsCache=True


Any advice? There are over 2000 Windows grids that use these same settings, so I would appreciate any advice on this before my next release.
(0035926)
UbitUmarov (administrator)
2019-12-03 11:08

<useLegacyJit enabled="1" /> should no longer be necessary

sure you meant 500kbps :p
(0035927)
aiaustin (developer)
2019-12-03 11:18
edited on: 2019-12-03 14:20

Ferd... if you still have these lines in... remove <useLegacyJit enabled="1" /> in files OpenSim.exe.config, OpenSim32.exe.config and Robust.exe.config in section RunTime in each. They are no longer needed in 0.9.1.0 or the latest dev master.

The extra line was added into those config files temporarily, and if anyone left those in they should now be removed.

(0035929)
gimisa (reporter)
2019-12-03 12:45

in opensim 091 useLegacyJit is not present in any of the above three files. OpenSim.exe.config, OpenSim32.exe.config and Robust.exe.config


In Opensim 091 release the the default setting presented in configuration file OpenSimDefaults.ini is the following :

    ; Support viewers object cache, default true
    ; users may need to reduce viewer bandwitdh if some prims or terrain parts fail to rez.
    ; change to false if you need to use old viewers that do not support this feature
    ;
    ; SupportViewerObjectsCache = trueSupportViewerObjectsCache = true in line

The setting above did not exist in 0901 OpenSimDefaults.ini which is probably an effort to cure the situation withness with prim lost .

Again as state by others changing bandwitdh of viewer does not seem to cure the situation I withness.
(0035930)
UbitUmarov (administrator)
2019-12-03 12:47

Support of Viewer Objects Cache is new on 0.9.1
Viewers had it for years, and we made no usage of it
(0035934)
tampa (reporter)
2019-12-03 23:04

https://i.imgur.com/e3bTay6.png [^] Logging in directly to the region, cached from previous login
https://i.imgur.com/bdYkylI.png [^] After 2 minutes of being unable to move around
https://i.imgur.com/opbpoxU.png [^] After 2 minutes after objects began to load, can move now

I sent this to Ubit already, think it is related to this maybe.

I have been getting mails about this for some time now, in total 6 different grids including two of my own. Stuff not rezzing, taking a long time after login for world to load etc. Just my two cents, is problem pls fix :)
(0035935)
gimisa (reporter)
2019-12-04 04:38
edited on: 2019-12-04 04:43

seem to work supportViewerObjectsCache = true for me.

Please note that Even if OpenSimDefaults.ini is the following :

    ; Support viewers object cache, default true
    ; users may need to reduce viewer bandwitdh if some prims or terrain parts fail to rez.
    ; change to false if you need to use old viewers that do not support this feature
    ;
    ; SupportViewerObjectsCache = true

Which would normaly provide for ClientStack.LindenUDP option SupportViewerObjectsCache = true but a config show request at consol level report nothing for ClientStack.LindenUDP,
One have to add it to gridCommun.ini or to opensim.ini as follow:

[ClientStack.LindenUDP]
SupportViewerObjectsCache=True

Region (root) # config show ClientStack.LindenUDP command in console does now report the setting proposed as follow :

[ClientStack.LindenUDP]
  SupportViewerObjectsCache = True

and that WORK for me hope it helped you too.
Thanks

(0035936)
UbitUmarov (administrator)
2019-12-04 04:54

config show only lists uncomented entries on ini files, not the full simulator settings/options.

The region code does define default values for all settings. no point on overloadind ini files( worse internal in memory structures created from them) setting the same again, as that command seemed to had assumed.
(0035937)
gimisa (reporter)
2019-12-04 12:02
edited on: 2019-12-04 12:03

Thanks for the clarification Ubit. As I see now the problem reappearing in same region with this setting on.

So that confirm your statement contrary to what I have said earlier in my enthusiasm about the setting change. I falsely reported that the problem was corrected for me where now I observe it again . In all probability the only thing that had help me seeing all the prims was likely cause I relog in the same region a second time around even with clear cache.

I appreciate your attention to this issue of 091 release.

(0035938)
UbitUmarov (administrator)
2019-12-04 12:08

I can't repo this issue. Been at tampa region.. no issues even with suicidal 3000kbps viewer bandwidth and 1024m view range, Either first time there, or returning
(0035940)
gimisa (reporter)
2019-12-05 03:27
edited on: 2019-12-05 03:30

Testing with cool and singularity viewers. Setting bandwidth to 500kbps and viewer reach to 128. Clear cache and logging in gimisa4 of 3d.gimisa.ca:9000 for three consecutive trials. Stats show 45ms ping time and no pocket lost. Result is all object appears properly after a loading time of about 3 minutes.

Repeating above procedure with maximum bandwidth (10,000 kbps ) three times .I got no pocket lost and same ping time as above. I got success of both viewer on first trial. Success of singularity on second and third trial with loading time of all objects within 1 minute. and failure to load all objects with cool viewer for the two other trials within 3 minutes of waiting time.

Firestorm is recommending 500kbps when using a wireless connextion ( my case ) . Ref: https://wiki.firestormviewer.org/fs_speedtest [^]

Thank you ubit for you help and attention .

(0035941)
UbitUmarov (administrator)
2019-12-05 05:40

Thx for you testing
(0035945)
alanscotch@yahoo.com (reporter)
2019-12-06 15:49
edited on: 2019-12-07 10:10

All of this has happened with a Firestorm Bandwith setting of 500 Kbps. (And Draw Distance of 600m)

Missing Terrain and objects (and AV Clothing grey) EVERY time I log in on 2x2 region

Except every time its different terrain , objects. But every time I login without previously deleting cache there inevitably is missing terrain.

Only fixed by deleting cache every time I'm on. ( then no terrain is missing next time I log in)

Only started after implementing OS 9.1.1

TPing to another region and back and the terrain and objects were restored.
I thot I had done that before and terrain had been still missing.
So I tried logging out and in 3 times ( without deleting cache) and TPing to another region and back – the terrain gets restored.

But all this time some clothing on my AV is grey ( and invariably remains grey ) but this ALWAYS gets fixed by deleting cache.

Firestorm 6.0.2 (56680) F (64bit)
 Welcome located at c-71-198-122-22.hsd1.ca.comcast.net (71.198.122.22:8009) SLURL: hop://wonder.outworldz.net:8002/Welcome/132/160/35 [^] (global coordinates 256,132.0, 256,160.0, 34.9)
 OpenSim 0.9.1.1 Yeti Dev (Win/.NET) Release Notes CPU: Intel(R) Core(TM) i7 CPU M 620 @ 2.67GHz (2660.01 MHz)
Memory: 8175 MB OS
Version: Microsoft Windows 7 SP1 64-bit (Build 7601)
Graphics Card Vendor: ATI Technologies Inc. Graphics Card: AMD Mobility Radeon HD 5800 Series
Windows Graphics Driver Version: 15.301.1901.0
OpenGL Version: 4.2.13417 Compatibility Profile Context 15.301.1901.0
libcurl Version: libcurl/7.54.1 OpenSSL/1.0.2l zlib/1.2.8 nghttp2/1.25.0 J2C Decoder Version: KDU v7.10.6
 Firestorm Viewer Skin: Firestorm (Grey)
Window size: 1920x1181 px
 Draw distance: 600 m
Bandwidth: 500 kbit/s
LOD factor: 2
Render quality: High-Ultra (6/7)
Advanced Lighting Model: Yes
Texture memory: 192 MB (1) VFS (cache) creation time (UTC): 2019-12-5T17:2:50 Built with MSVC version 1800
Packets Lost: 16/58,690 (0.0%) December 05 2019 17:40:41 SLT
Reported In:Firestorm 6.0.2.56680 Animesh Release

(0035946)
gimisa (reporter)
2019-12-06 18:27

I went to Alen location use cut past in local chat of your viewer and clic the following link. secondlife:///app/teleport/wonder.outworldz.net|8002|Welcome/132/160/35 [^] .

No previous clear cache 500kbps 128m reach distance, my ping was 115ms , no pocket lost reported. I waited 5 minutes to make sure everything was downloaded . I witness that terrain portion is missing and obvious terrain texture is not appearing.

Please see https://pasteboard.co/IK54v3d.png [^] for representation of my viewer screen.

I suggest other to report their scene rendering problem here so Ubit can see a pattern that he can reproduce.

Hope it help
(0035947)
gimisa (reporter)
2019-12-07 04:42
edited on: 2019-12-07 04:47

New test and a good news ( so far ) . I wanted to have a consistent repeatable failure test for Ubit. Also need a mean to measure the result. So I set my singularity viewer to 5000kpbs and 128m range. To measure the result I use the stat window advance display with a paid attention to the object count number. That number is increasing as long as the new object count per second is more then 0 .

The following link give you the result of 4 consecutive trials with clear cache between each. The result is that missing prim would happen 2 times out of the 4 trials. https://pasteboard.co/IK8UCgt.png [^] Object count stabilise at 19978 when new object count is 0 and fail to meet that value when prim are missing.

Now the good news. I went to osgrid and talk to ppl that told me they dont have that issue , either they using the previous release of osgrid bin or the newest one. The previous one OpenSim 0.9.1.0 Snail Dev OSgrid 0.9.1.0 695d807696: 2019-01-26 and the new one is http://danbanner.onikenkon.com/osgrid/osgrid-opensim-11012019.v0.9.1.26b5aff.zip [^] .

So I installed the newest release put my own grid ini settings in and run the simulator for gimisa4 again. This time 4 trials 4 success...No missing prim on land. Total object count result is a bit higher at 19995 when new object count stabilize to 0 at every of the 4 trials without object missing. So this proves (so far ) that bin of official snail release as an issue which next Yeti version seem to address.

I will leave gimisa4 region with this installation if you like to visit cut past and clic in local chat secondlife:///app/teleport/3d.gimisa.ca|9000|gimisa4 [^]

Hope it helps.

(0035948)
alanscotch@yahoo.com (reporter)
2019-12-07 11:02

Thanks gimisa --

All of the following was done with Draw distance: 600 m and without clearing cache:-

Here's another ( better?) workaround ( for me):-
Setting Bandwidth to 400 kbit/s , positioning my AV at two different corners of the 2x2 and logging in and out 5 times -- there was NEVER any loss of terrain and my AV was never grey !!!
I then changed it back to 500 and logged in and out twice -- both times there was loss of terrain .
And then back again to 400 with 2 logins -- no loss of terrain.
And I don't need to clear cache to stop my AV from being grey.

Other tests:-
I logged in and out 6 times at Bandwidth: 1500 kbit/s ( instead of 500) and only half the time there was a loss of land.
I put it back to 500 and changed DRAW DISTANCE to 200 ( instead of 600) positioned my self at 2 different corners of the 2x2 and logged in 4 times -- every time there was loss of terrain.
Each of these losses still gets fixed by TPing to another region and back ( but it never fixes my grey AV - only clearing cache does that).
Changing DRAW DISTANCE to 800 made no difference.
(0035949)
alanscotch@yahoo.com (reporter)
2019-12-08 08:54
edited on: 2019-12-08 12:55

But then I logged in today ( the next day) and there was loss of terrain.
 So I set it at 300 and the 2nd time I logged in there was loss of terrain.
Its now at 200 and after 3 logins -- no loss.
See how it is tomorrow :(

( What's the drawback side-effect of 200 ? )

(0035950)
UbitUmarov (administrator)
2019-12-08 08:58

been there a few times, even with 3000kbps no issues :(
(0035951)
UbitUmarov (administrator)
2019-12-08 09:01

you are getting in region by direct login or tp from other?
(0035952)
alanscotch@yahoo.com (reporter)
2019-12-08 12:56
edited on: 2019-12-08 15:33

Direct login
( TP to another region and back always fixes it -- but not my grey AV)
All of my testing has been on the 2x2 region "Winter" not the 1x1 "Welcome" ( as my visitor detector tells me )

I noticed the omission that my server has an SSD -- no HD -- super fast

(0035953)
gimisa (reporter)
2019-12-08 20:13

Well sorry to report that I witness missing prim in gimisa4 with 0911 release and 5000kbps. Turning it back to 500kbps and re loging in same region still missing item with 0911 from osgrid. All prim show on tp from an other region not adjacent to gimisa4.
(0035954)
UbitUmarov (administrator)
2019-12-09 03:03

issues are to be expected with highband width, viewers just can't handle it, that is why i mentioned "even with 3000kbps"
(0035955)
aiaustin (developer)
2019-12-09 07:31
edited on: 2019-12-09 07:48

gimisa and Alan... assuming you are using latest 0.9.1.1 dev master (or the OSGrid latest update which is also latest dev master code), can you try a viewer based on latest LL code base 6.3.4 to see if that has any effect?

Unfortunately Firestorm for OpenSim is not yet updated and is still at version 6.0.2. There are experimental builds but they are not yet openly released so it is best not to try those for this test.

But most other TPVs are now updated to include LL 6.3.4 code... e.g.

1. Get the up to date Singularity installer (1.8.7 (7992) 24-Nov-2019) from the bottom at this page:
https://build.alchemyviewer.org/browse/SV-BETA/latestSuccessful [^]

2. or use Cool VL Viewer latest version 1.26.24 (2019-11-30) from the downloads link at the top of:
http://sldev.free.fr/ [^]

(0035956)
gimisa (reporter)
2019-12-09 09:04
edited on: 2019-12-09 09:08

Hi Sir I thank you for your attention. I running under ubuntu 18.04.3 up to date so I cannot use the pre compiled version offered for window. If you read my earlier post you will see that I am testing with singularity and cool viewer as follow:

    "I use viewer singularity alpha issue 6994 dating 2018-04-9 . I have been using this viewer since release. I also tried cool viewer, 1.26.4 this week release reading above that the issue was possibly with firestorm.
    It is interesting to note that I have one computer that I could not update cause too old. That computer runs ubuntu 12.04 with mono version 3.2.8 and gimisa3 region simulator version 0901 official release. The rendering there is without issue. "


Its hard to validate if test is conclusive without visiting the complete region attentively when only a few prim are missing. So I have use a test criteria with the help of the statistic window advance stat section total objs count . see https://pasteboard.co/IK8UCgt.png [^] Looking for consistency in reported object count in my case 19990 objects in this regular region.

Aside from opensim offical release I have also tried with osgrid latest http://danbanner.onikenkon.com/osgrid/osgrid-opensim-11012019.v0.9.1.26b5aff.zip [^] [^] . I withnessed same failure after a time at 500kbps. Its hard to get consistent failure most of the time its good and once of a sudden it fails.

I am now testing Lanani latest. https://mosthugs.win/download/N3December2019opensim-0.9.1%20Binary.zip [^] She seem to have regressed her last release ( november ) in her december release to something that in her experience works well. With it so far I notice that at 500kbps bandwidth everything is fine . Crossing fingers ...

Hope the above report helps .
Thanks again for your and Ubit works to make this new release a success.

(0035957)
alanscotch@yahoo.com (reporter)
2019-12-09 09:06

Installed Singularity Beta 1.8.7 . It (eventually) defaults to Max Bandwidth of 750. (I set Draw Distance to 608 and Graphics ULTRA)
At 750 there was terrain loss -- at 500 too -- at 200 no loss
(After first installing Singularity Max Bandith was at 2000 -- somehow it self adjusted to 750.)
First time I TP'd to another region and TPing back the Viewer crashed ( second & 3rd time after relogins - no crash)
Singularity has another setting TEXTURE BANDWIDTH which defaults to 3000 which I left alone.
PS I hate that I can't resize Camera Control and there's no Mouselook button !
(0035958)
UbitUmarov (administrator)
2019-12-10 04:24

Hi, you both run the viewers in linux or just Gimisa?
(0035959)
gimisa (reporter)
2019-12-10 05:11
edited on: 2019-12-10 11:50

For me as stated its ubuntu 18.04.3 for Alan you have your reply above when he fist posted .

" Version: Microsoft Windows 7 SP1 64-bit (Build 7601) "

 So far Lanani build does not create the issue with me at 500kbps. She offers the source if you want to compare. Please note that in official release 0901 bandwidth setting was never an issue at least to me. As stated by Alan the default setting of singularity is 750kbps . Most user will not touch that and will simply consider that the actual region they are visiting is lagging and move on ....

Hope it helps .

(0035960)
UbitUmarov (administrator)
2019-12-10 05:16
edited on: 2019-12-10 11:23

We use new viewers features (many years old in them), like local object cache. That is different protocol, so different issues.
We do know that high bandwidth set always causes these issues,
Thing is why only some have problems now :(

(0035961)
alanscotch@yahoo.com (reporter)
2019-12-10 11:29
edited on: 2019-12-10 17:22

I just lost some terrain at 200 ! This sucks ( so far no grey AV)

So I set it back to 500 and logged out and in to a different 2x2 region and there was lost terrain. So its not just happening in one Region (2x2).

(0035962)
UbitUmarov (administrator)
2019-12-10 11:35

and other regions? like osgrid lbsa, wright plaza etc ?
(0035964)
gimisa (reporter)
2019-12-11 03:52

So your recommendation is use window 10 with new viewer and set bandwidth to 500kbps.

Can I suggest adding this in the release notes as minimum requirements for opensimulator version 091 .

Thanks for clarifying.
(0035965)
UbitUmarov (administrator)
2019-12-11 05:20
edited on: 2019-12-11 06:35

I recommended nothing.
Cannot make recomendations like that, without understandig the issue.

Some I can, at least to test
- run only one viewer at a time.
- keep it the selected application...

Run it with the default settings about region objects caching.

(0035966)
aiaustin (developer)
2019-12-11 06:32
edited on: 2019-12-11 06:33

As Ubit said. I don't think he made such restrictive recommendation anywhere gimisa.

It remains to be seen what the issue is, but we all do appreciate it is happening. We did observe this a lot during development, and indeed I was the initial reporter on this issue back in April 2019. See attached images for typical visual glitches seen then. But those issues were addressed some time ago and even with excessive bandwidth settings in testing some of us are not now seeing the terrain and object loss at all.

Lets hope we can pin down the common aspects of the environment or context that cause this one day.

(0035969)
tampa (reporter)
2019-12-11 06:43

Since I turned off object cache I receive reports that previous issues, since the upgrade to master code, are resolved and objects rezz and behave properly. I know that's not exactly evidence, but you cannot help but wonder what's going on there. I have one last grid with defaults, they just reported in 2 more problems regarding objects and specifically mesh not showing...

I also find it sad that we seemingly lower the bandwidth each major version instead of aiming to use the new push in broadband connectivity to actually support +1500kbps. It feels like going backwards instead of forwards.
(0035970)
gimisa (reporter)
2019-12-11 07:53

I agree lets hope we can pin down what context would reliably make it work. As you know oscc is being held this week-end with opensimulator release 091 . The question will be is what I see what you see?

I though to give a trial to OSCC keynote 2 which is available for visiting as we speak. Using the object count as criteria here is the three test results I did there.

Clear cache before each test. Used cool viewer. Set at 500kbps and reach of 128m. Teleport in same location, look in same direction. Result taken after object count stabilized. All three reference pictures look similar. Detail can be found in picture as follows:

Window 10 , object count 2813, https://pasteboard.co/IKLSeRE.png [^]
ubuntu 18.04 , object count 3601, https://pasteboard.co/IKM2URY.png [^]
ubuntu 14.04 , object count 2354, https://pasteboard.co/IKLR9L7.png [^]
(0035971)
aiaustin (developer)
2019-12-11 08:00
edited on: 2019-12-11 08:00

The issues occurred previously with SupportViewerObjectsCache set to false and true as reported above. The default for this setting now is true. Ubit indicated the mechanism changed even if you have SupportViewerObjectsCache set to false.

Note I am running with Firestorm set to 1500bps. I appreciate our environments and the regions/grid we are trying all vary of course. One day I am sure we will pin this down! Yours optimistically.

(0035972)
aiaustin (developer)
2019-12-11 08:07
edited on: 2019-12-11 08:19

gimisa… just a thought... check your Winver with the command "winver". It looks from the pasted info that you are reporting your Windows 10 version is a really old one...

1511 10586 November 10, 2015

There have been many fixes and .NET updates since then. I wonder if you have a test environment with the latest Windows updates?

1909 18363 November 12, 2019

(0035973)
aiaustin (developer)
2019-12-11 08:22
edited on: 2019-12-11 08:29

The issue of mesh not showing until you click on it is a different problem. This issue is for terrain lumps missing or objects (prims and/or mesh) not showing and no amount of waiting or clicking where it should be shows the outline or makes it rez. Lets try to separate these issues to home in on the problem. Anyone reporting mesh not showing please ask them if it shows an outline or rezzes when they click where it should be. If so lets separate that issue out.

(0035974)
tampa (reporter)
2019-12-11 09:24

There were also changes to sending terrain patches based on viewer view distance setting, I actually planned to revert that commit after seeing nearby terrain tiles not loaded properly, but reproducing this is hard as the failures depend on the protocol itself, which is kinda meh.

Frankly something seems botched with viewerobjectcache and those view distance changes and I think the best test-case really is reverting those and trying one-for-one to see if the problems can be reproduced, I don't see another way.
(0035975)
beqjanus (reporter)
2019-12-11 10:54

Hi all,
I am reviewing https://jira.firestormviewer.org/browse/FIRE-29196 [^]
It would seem that at present there is no clear suggestion that this is viewer related, especially if this is new behaviour. I am going to leave the issue open, as this discussion is clearly ongoing, but for now, I'll put it in the "probably not viewer" pile.
(0035976)
gimisa (reporter)
2019-12-11 13:40
edited on: 2019-12-12 01:50

winver: https://pasteboard.co/IKOl49C.png [^]

So far I only validate if I see or not see the object. I did not click on it to validate if the mesh is there.

But I know what you mean. I have seen that happen before when you try to rez new imported mesh . You see a frame of it when you click on where it should be even if you dont see the object.

I will be in the look out to that fact next time I witness missing one.

(0035978)
aiaustin (developer)
2019-12-12 01:50
edited on: 2019-12-12 01:51

Gimisa.. as you noted in the image, your Windows 10 version is 1809 (issued November 13, 2018) so about a year old and pretty recent. There have been two updates since then to 1903 and 1909. I have no information that this will be relevant, but in the interests of pinning this down I thought I would check.

https://en.wikipedia.org/wiki/Windows_10_version_history [^]

(0035980)
gimisa (reporter)
2019-12-12 09:03

Very interesting , I though window 10 was updating automatically . I request an update via parameter command to comply with your request and saw only little updates to net for exemple. I got restart request . When back in I did winver command as per your earlier instructions and got the SAME version number.!!

See for yourself: https://pasteboard.co/IKVJdLz.png [^]

I check with :
"Select the Start button > Settings > System > About ."

And got same value. Additional detail provide with above function is installation date 2019-11-09 ( this computer is brand new DELL )

Here is more detail of Net and C++ version number should that be helpfull: https://pasteboard.co/IKW1iiR.png [^]

Clear cache and going back to oscc keynote2 but this time with default cool viewer bandwidth of 2000kbps I got 2865 object count when stabilised compare to 2813 on earlier report .

Hope it helps
(0035981)
alanscotch@yahoo.com (reporter)
2019-12-12 11:26

I just logged in again at 200 and for the first time at 200 my AV skin was grey ( and some terrain was missing) -- So I'm not sure how DIRECTLY involved MAX BANDWIDTH is in all of this? ( Other than it happens less often the lower the Bandwidth). But clearing CACHE always works.
(0035986)
aiaustin (developer)
2019-12-12 14:44
edited on: 2019-12-13 01:15

I had a couple of machines where the automatic update did not get offered... I used the manual update assistant for those...

https://support.microsoft.com/en-gb/help/4028685/windows-10-get-the-update [^]

Though, I have no evidence the version you are using is an issue gimisa.

(0035988)
UbitUmarov (administrator)
2019-12-13 09:38

sorry for asking such thing, but can you test the exact same viewer at SL ?
(0035989)
gimisa (reporter)
2019-12-13 12:44
edited on: 2019-12-14 06:08

Two questions to be replied.

The first one is from Mister Austin as follows:
...This issue is for terrain lumps missing or objects (prims and/or mesh) not showing and no amount of waiting or clicking where it should be shows the outline or makes it rez.
To concure with this statement following two pictures.

One show 2 chairs and machine with missing one in the middle in edit mode. Is can be seen its totaly not there compare to second picture showing all of them in place in edit mode for comparaison.
Missing chairs: https://pasteboard.co/IL6MRez.png [^] Not missing chairs: https://pasteboard.co/IL6OMYL.png [^]

The second question could we get same experiment with same viewer ( and settings 128m reach distance and 500kbps ) .

So this experiment is in SL Svarga location , a location that includes only prims not even sculpty.
Since I dont know the region enough to spot any missing thing all I can do is report the number of objects statistic presents.
 
First trial no viewer action simply rez in and correct for avatar white out by editing apparence.
Second trial is with clear cache and rez back in svarga same position same looking direction.
Third trial is no clear cache rezzing again in svarga.
 
1- 5120 objects reported https://pasteboard.co/IL6QqO2.png [^] .
2- 3876 objects reported https://pasteboard.co/IL6SqHR.png [^]
3- 4259 objects reported https://pasteboard.co/IL6TdfO.png [^]

A day after this test this is what I saw with same setting in a visit to sunset.outworldz.net:8002 everything seem normal except a house I happen to check out . https://pasteboard.co/ILcQ8ZA.png [^] .

For me it seem related to time somehow like its harder the first logging of the day in a region when I had no visitor. Can it be connected to flotsamcache the cache module as this one has a schedule for cleaning up. I will diseable this module later and see if it has influence on tomorrow result.

hope it helps

(0035990)
alanscotch@yahoo.com (reporter)
2019-12-14 12:35
edited on: 2019-12-14 19:55

I probably should disclose that both the server desktop and the laptop running the viewer enjoy Comcast's 40 Mbps upload and 10 download ( as well as everything on the server being on an SSD - no HD ).

Also the rez point on the tested 2x2 region was mostly 10m away from the edge of the region. Any terrain loss seemed to be always over 256m away.
Yet terrain loss is invariably associated with a grey AV.

(0035991)
gimisa (reporter)
2019-12-14 13:31

During oscc19 on Dec 14 at 13:00 PST this is what Caro Kelso and me were seeing from the stage of "Other Social Worlds and the Future of the Metaverse" presentation :

gimisa nothing https://pasteboard.co/ILgt6YC.png [^] my viewe
kelso 3 seats 4 avatar no hat view https://pasteboard.co/ILgtJQa.png [^]
caro 3 seats 4 avatar with Mel with Hat https://pasteboard.co/ILgA9Hn.png [^]
(0035992)
aiaustin (developer)
2019-12-14 14:33
edited on: 2019-12-14 14:34

Just to report that I was using Firestorm 6.3.5 (experimental build for tests) at the OSCC19 opening plenary session, core dev panel and a following talk with 1500bps set for bandwidth, ultra display setting and 512m view distances and had no terrain or object loss. Ditto when I visited the shopping plaza and several of the expo areas.

(0035993)
tampa (reporter)
2019-12-14 14:56

oscc is a poor metric to test this on, so is OSGrid, there are too many variables at play there that, due to changes on how this stuff is sent in the first place, make it hard to determine if it is just a network glitch or an actual bug. Remember this stuff sends over UDP, even with the protocol as is it is difficult to determine if stuff has sent or not.

Since the problem occurs as of recent changes, what do we actually know here? We know the recent changes caused problems, so naturally they have to be the cause right? I am not so sure, I think the real cause here is an age old protocol deficiency that the recent "fixes" have brought to light.

As such I went back and looked at the changes in the commits and I reverted some of them. Now I found the protocol itself to be more stable, more consistent udp traffic and no more "burst and fails" as I have been seeing.

I get the "send by view distance" is a design to actually reduce the traffic and only send what is needed, but I think viewers cannot handle these bursts of data is creates. At least the Firestorm logs suggest udp pipeline breakdowns no matter the bandwidth.

And again, should the thrive not be to have viewers and OpenSim stable at ever higher bandwidth settings. Folks we are not in 2010 anymore, most people have more than 2mbit in download speeds if not more. Sure, adjust for those who don't, but at the same time don't cripple what capabilities we have just because of that.

I may make a patch of my changes at some point which both reverted and changed some stuff, which so far have resolved most issues I was made aware of by now over 10 different grids.

I strongly urge to review the changes to view distance rendering and sending of data as well as viewer object caching as those, for me and at least 6 other grids, were the main cause of trouble with master code. With that many people reporting issues and now a return to the status quo I simply find it hard to blame anything but those changes.
(0035994)
gimisa (reporter)
2019-12-14 15:23
edited on: 2019-12-14 15:27

That post dating 2010 already state that bandwidth was not an issue back then when internet download speed was in the order of one or two mega ... Thanks to Shelenn to have point that one out for me:

http://joelfoner.com/2010/03/understanding-the-maximum-bandwidth-option-in-the-second-life-viewer/ [^]

So more then eager to test what you did Tampa

(0035995)
aiaustin (developer)
2019-12-15 01:11

Good work tampa. I hope this lets us home in on the problems. I have no doubt that the terrain loss and missing objects was first seen, and then quite seriously, in early April 2019 and was mitigated but jit fully resolved after that.

On the changes in bandwidth for users you note, I would point out though that some users, including me, will have been using systems with very fast gigabit university connections for their viewers for some years in SL and OpenSim without such problems showing.
(0035996)
aiaustin (developer)
2019-12-15 01:21
edited on: 2019-12-15 01:22

Understanding a bit more about how the viewer side region cache actually works might help, but that may be inappropriate for this already long thread of comments on the issue. I do see quite a bit of white where textures should be and always fix that by clearing the cache. I use my viewers for SL and multiple OpenSim grids with direct login or HG jumps, will be changing view distances, zooming way out and back (way beyond the view distances) and in testing often am teleporting all over the place and will often not have a full region, so have bot worried that this may be a serious issue, but it does happen multiple times each week.

So this makes me wonder how robust the viewer side region cache mechanism is... especially in three situations...

1. revisiting a region after an earlier visit in same viewer session where the whole content was nit downloaded before avatar moved away.

2. having multiple avatars log in with multiple instances of the same viewer on one system. I often in testing have 3 or 4 viewers running.

3. changes made by one user in a region while other users are logged in being properly propagated.

etc. !!!

(0035999)
UbitUmarov (administrator)
2019-12-19 03:14

Made a few changes that may have impact on this
(0036000)
aiaustin (developer)
2019-12-19 06:06

I saw the changes. Openvue and Ailand grids (all regions) and OSGrid "RuthAndRoth" region all now on latest dev master with the UDP ack and resend changes (opensim-0.9.1.1Dev-69-g066a6fb.zip).

I am not myself seeing the terrain and object loss recently, but the update is in place in case others wish to test from their own viewer environments.
(0036001)
gimisa (reporter)
2019-12-19 07:55

Hi would like to thank Ubit for the work he is doing on this difficult problem .

Per my test I witness that its hard to tell if viewer is effectively presenting all of region object if one does not know the region. There could be a wall or other object masking the missing object. That is why my test include a numeric value corresponding to the object counted in the scene by the viewer and in condition at hand.

The important point is consistency in reported value after proper time is allowed to let the count stabilize. One might question this test. I build confidence in this test by doing it in gimisa3 gimisaOS welcome region. This region runs "OLD" opensim release 0901 . The three following pictures shows a number of objects consistent after each loging . The first loging is after visiting virtual.aiai.ed.ac.uk:8002 which I will present results next. Second test is after clear cache . Third test is just reloging after second test.

gimisa3 test 1 object count 9309: https://pasteboard.co/ILZxLoJ.png [^]
gimisa3 test 2 object count 9320: https://pasteboard.co/ILZy1uYR.png [^]
gimisa3 test 3 object count 9300: https://pasteboard.co/ILZyf4i.png [^]

Visiting virtual.aiai.ed.ac.uk:8002 welcome with same condition first teleport without doing anything . Second teleport after clear cache before logging , third teleport after doing nothing.

welcome test 1 object count 2376: https://pasteboard.co/ILZysIg.png [^]
welcome test 2 object count 2557: https://pasteboard.co/ILZyEM9.png [^]
welcome test 3 object count 2441: https://pasteboard.co/ILZyT9Z.png [^]

Please note the difference in prim count ( a larger test in my region ) and in consistency of results . I am only a user and people with more knowledge could comment on what the viewer is reporting when object count is presented and if truly pertinent to the situation being tested.

As to its results your conclusion is as good as mine...
(0036002)
aiaustin (developer)
2019-12-19 08:49
edited on: 2019-12-19 13:56

Using Firestorm 6.0.2.56680 with 1,500kbps bandwith and 512m draw distance
Openvue grid, Openvue region, Welcome plot (which covers whole 256m256m region) has 1,789 prims.

hop://virtual.aiai.ed.ac.uk:8002/Openvue/128/128/29 [^]

Firestorm 6.0.2.56680 (Feb 9, 2019) Objects - land impact = 1,789.
Scene Load Statistics Object Count = 1,938.

Tested also with Singularity Beta 1_8_7_7997_x86_64, using 512m viewer distance and 2,000kbps bandwidth and 3,000kbps texture bandwidth...

Singularity Beta 1_8_7_7997_x86_64 (Dec 13, 2019) Primitives on parcel: 1,789
Scene Load Statistics Object Count = 1,938.

The higher scene load statistics object count versus the reported prim/land impact is expected as Ubit notes below.

(0036003)
aiaustin (developer)
2019-12-19 09:02
edited on: 2019-12-19 12:06

Using Firestorm 6.0.2.56680 with 1,500bps bandwith and 512m draw distance.

Visiting 3d.gimisa.ca:9000 gimisa3 I see a region of 256x256 and the plot name "gimisa3 WELCOME region" which is not the full region that contains 5,175 prims/land impact equivalents. Note that the plot area is 60,512sq.m. So there are other plots on the region. I see 3 other plots when I use "Show property Lines". A total of 7,210 objects are in use on the region across all plots. Scene Load Statistics Object Count = 7,375.

Visiting 3d.gimisa.ca:9000 gimisa4 I see a region of 256x256 and the plot name "gimisa4 Castle of ODDS" which is the full region in size (65,536sq.m) that contains 10,973 prims/land impact equivalents. Scene Load Statistics Object Count = 11,122.

The higher scene load statistics object count versus the reported prim/land impact is expected as Ubit notes below.

(0036004)
aiaustin (developer)
2019-12-19 09:11
edited on: 2019-12-25 02:40

Using Firestorm 6.0.2.56680 with 1,500kbps bandwith and 512m draw distance.

Visiting Alan's wonder.outworldz.net:8002 grid

Welcome is a 256mx256m region with one land plot (named as the default "Your Parcel") and contains 1,725 prims/land impact. Scene Load Statistics Object Count = 2,115.

Winter is a 2x2 (512mx512m) var region with one land plot "Winter 2X2" that reports 4,080 prims/land impact. Scene Load Statistics Objects Count = 4,939.

The higher scene load statistics object count versus the reported prim/land impact is expected as Ubit notes below.

(0036005)
UbitUmarov (administrator)
2019-12-19 09:19

Viewer stats objects counts include things like each terrain 16x16m terrain patch etc
(0036006)
aiaustin (developer)
2019-12-19 13:45
edited on: 2019-12-20 13:05

gimisa, on your images of the three visits to Openvue..where do you get the (different) object counts from?

In each image, as far as I can see it says objects = 1,805.

I was not sure why Cool VL reports different numbers to Firestorm and Singularity, so I installed the latest CoolVLViewer-1.26.24.4-Windows-x86_64-Setup.exe (2019-12-14) to test myself... using default (?) bandwidth of 2,000 kbps, view distance = 512m...

hop://virtual.aiai.ed.ac.uk:8002/Openvue/128/128/29 [^]

About land - Objects - Primitives on parcel = 1,789.
Simulator Statistics Object Count = 1,938.

I.e. identical to Firestorm and Singularity.

(0036007)
gimisa (reporter)
2019-12-24 06:20

Hi M. Austin. I thank you for your question.

I pretend that a simple examination of the land when you rez is not sufficient to asses the situation in regard to invisible prims.

I propose to compares object count from viewer statistic provide after stabilization for multiple logging to validate if the object presented to the user is all the object to be presented. So when count is similar between multiple logging I feel confident that all the object are presented by viewer and visible to user.

Please note in region where you have child agent created by the simulator in adjacent region the object count will be MUCH more important then the land count result by the amount of object visible to the viewer from those region. To be comparable one must be located and oriented in same direction on each logging.

Might I take this occasion to wish all opensim dedicate developers a productive 2020 for opensim.
(0036008)
aiaustin (developer)
2019-12-24 06:55

Firestorm Viewer advice on setting bandwidth....
https://wiki.firestormviewer.org/fs_speedtest [^]
(0036009)
gimisa (reporter)
2019-12-24 10:39

Sir,

Above firestormviewer web link is same as I posted in this mantis post (0035940)
gimisa (reporter) 2019-12-05 03:27 thanks for re posting.

Might I re post also the (0036001) done 2019-12-19 07:55 Mainly its results. Understanding that all three picture on each region were taken logging repeatedly in the region of test and reporting viewer statistic object count on each of the three logging.


gimisa3 test 1 object count 9309: https://pasteboard.co/ILZxLoJ.png [^] [^]
gimisa3 test 2 object count 9320: https://pasteboard.co/ILZy1uYR.png [^] [^]
gimisa3 test 3 object count 9300: https://pasteboard.co/ILZyf4i.png [^] [^]


welcome test 1 object count 2376: https://pasteboard.co/ILZysIg.png [^] [^]
welcome test 2 object count 2557: https://pasteboard.co/ILZyEM9.png [^] [^]
welcome test 3 object count 2441: https://pasteboard.co/ILZyT9Z.png [^] [^]

You might be able to do similar test and come to your own conclusions.
(0036010)
BillBlight (developer)
2019-12-24 10:51

Trying to figure out where you got those object counts gimisa ..

All three of your images in each set show the same number of actual objects ..

i.e. https://i.gyazo.com/193f44cd81e2c1f00ba79891723d5e17.jpg [^]
(0036011)
aiaustin (developer)
2019-12-24 12:54
edited on: 2019-12-26 01:50

Me too.

(0036012)
mike.dickson (reporter)
2019-12-26 06:00

The latest changes don't fix the issues I see. I do think it can be related to a client with less than perfect connectivity. I say it that way because with http and the udp stack in the server its supposed to handle these cases and not lose data. I dont see tons of retransmissions but at times a backup in un-ack'd data. Afain the protocols should handle that and not fail to rez objects or terrain.

Tampa back on 12/14 alluded to having backed out some change-sets and as a result getting better performance. Any chance you could share??

One more observation. With bandwidth set to 1500 on a hard wired connection I see the available bandwidth from "show queues" as closer to 2400. IDK if that number is calculated somehow but I've never seen it match the viewer settings for desired bandwidth. Might be something to that.
(0036013)
gimisa (reporter)
2019-12-26 12:09

Thanks for the question. There is a few object count in stat its true.

I believe the important stat in our test is the render object count found in the advance pane of the viewer statistic .

 Let me post back the image presenting 4 stats on gimisa4 that I posted
(0035947) 2019-12-07 04:42 https://pasteboard.co/IK8UCgt.png. [^]

To obtain this image I copied 4 time the result object count presented in advance render object count as presented here: https://pasteboard.co/IN5f1Od.png [^]

hope it helps
(0036019)
aiaustin (developer)
2019-12-30 10:08
edited on: 2019-12-31 06:19

gimisa and alanscotch… we are trying things again and I was just visiting 3d.gimisa.ca:9000 gimisa4.
It is on OpenSim 0.9.1.0 Snail Release (Unix/Mono)

It all looks good to me even on a very slow link away from my usual rig this weekend.

I can't reach http://wonder.outworldz.net:8002 [^] Welcome today.. but I know Alan said he was already on latest (recent anyway) 0.9.1.1 dev code.

Ubit did make a change to UDP handling to help prevent overload of data going to the viewer when it gets overloaded.. We know there are a lot of objects (10K+ on gimisa4), and in testing cache clearance and so on all can add temporarily to the load. I know you let things settle before reading off object numbers.

To try to get a handle on this I wonder if its possible for you (both?) to

a) update to latest 0.9.1.1 dev master as at 2019-12-18 23:26 which incorporates the UDP throttles. Any adjacent regions that will be within view distance ought to be updated too just as belt and braces.

b) use a viewer that we know how it works (or think we do?) try Firestorm 6.0.2 if you will until we get to the bottom of the problem.

c) log on and let your inventory completely load,

d) then do a few log on tests and look round. Ideally test on a region that is placed a couple of regions away from an adjacent region (assuming you don't set the view distance larger than 512m) so that there is less confusion with adjacent region issues.

If you see especially terrain chunks missing that will be helpful to see the snapshot. I think object numbers are likely to be confusing and differ by lots of things, but if you definitely know there should be an object somewhere and cannot see it, and if you click in the area its meant to be and no outline wireframe or object shows when you do that... that is what we want to know too.

(0036022)
aiaustin (developer)
2019-12-31 14:58
edited on: 2020-01-01 06:30

gimisa reports he is fixed after update of his grid to latest dev master as at 18-Dec-2019.

AlanScotch... if you use DreamGrid , can you update to 3.299 which now includes the 18-Dec-2019 OpenSim 0.9.1.1 dev master incorporating the UDP throttle change and report any issue related to this Mantis after that?

(0036023)
mike.dickson (reporter)
2019-12-31 15:02

I can safely say that I haven't seen terrain drops with the new code but this is definitely not fully resolved with prims failing to rez. UDP is still problematic especially with anything like a lossy connection.
(0036024)
aiaustin (developer)
2019-12-31 15:14

Mike, do you mean completely missing even if you click where they are meant to be, or do they show a wireframe or appear when the area they should be in is clicked?

Which grid and region, and what viewer version if so? Lets try to pin this down, I am on a nice slow connection with packet loss showing for next few days!
(0036025)
gimisa (reporter)
2020-01-02 04:07

M. Aiaustin here is results of test done with dec 18 build : http://opensimulator.org/dist/OpenSim-LastAutoBuild.zip [^]

First image compare Firestorm versions 64_5.0.7.52912, 64_5.0.7.6.0.2.56680 and singularity 1.8.7.6994 . Second page show scenegate 0_1_2_41405 the new viewer release and cool viewer 1.26.24 .

All test were done three time one as is , on with clear cache and one as is again. Also except for the test done with the newest firestorm version all other test were done so they compare with previous notes posted in this mantis.

Firestorm-Singularity show success : https://pasteboard.co/IO66FD0.jpg [^]
Scenegate-coolViewer show problems : https://pasteboard.co/IO68PpP.jpg [^]


Hope it helps.

- Issue History
Date Modified Username Field Change
2019-04-09 03:31 aiaustin New Issue
2019-04-09 03:31 aiaustin Status new => assigned
2019-04-09 03:31 aiaustin Assigned To => UbitUmarov
2019-04-09 03:33 aiaustin Relationship added related to 0008515
2019-04-09 03:35 aiaustin Additional Information Updated View Revisions
2019-04-09 08:07 aiaustin Note Added: 0035073
2019-04-09 08:09 aiaustin File Added: 2019-04-09-Terrain-Loss-1.jpg
2019-04-09 08:09 aiaustin File Deleted: 2019-04-09-Terrain-Loss-1.jpg
2019-04-09 08:10 aiaustin File Added: 2019-04-09-Terrain-Loss-1.jpg
2019-04-09 08:21 aiaustin Note Added: 0035074
2019-04-09 08:22 aiaustin File Added: 2019-04-09-Prim-Loss-1.jpg
2019-04-09 08:23 aiaustin Note Edited: 0035074 View Revisions
2019-04-09 08:45 UbitUmarov Note Added: 0035075
2019-04-09 08:47 aiaustin Note Added: 0035076
2019-04-09 08:53 aiaustin Note Edited: 0035076 View Revisions
2019-04-09 22:03 mewtwo0641 Note Added: 0035077
2019-04-10 02:04 aiaustin Note Added: 0035078
2019-04-10 02:04 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:08 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:10 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:12 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:13 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:16 aiaustin Note Added: 0035079
2019-04-10 02:19 aiaustin Note Edited: 0035079 View Revisions
2019-04-10 09:09 aiaustin Note Added: 0035080
2019-04-10 09:10 aiaustin Note Edited: 0035080 View Revisions
2019-04-10 09:19 UbitUmarov Note Added: 0035081
2019-04-10 10:03 aiaustin Note Added: 0035082
2019-04-10 10:19 BillBlight Note Added: 0035083
2019-04-10 11:03 aiaustin Note Added: 0035084
2019-04-11 09:41 aiaustin Note Edited: 0035081 View Revisions
2019-04-13 01:00 aiaustin Note Added: 0035085
2019-04-13 01:01 aiaustin Note Edited: 0035085 View Revisions
2019-04-13 06:25 BillBlight Note Added: 0035086
2019-04-13 09:57 aiaustin Note Added: 0035092
2019-04-13 10:07 aiaustin Note Edited: 0035092 View Revisions
2019-04-13 10:08 aiaustin Note Edited: 0035092 View Revisions
2019-05-14 09:06 aiaustin Relationship added related to 0008528
2019-06-05 11:19 aiaustin Note Added: 0035288
2019-06-11 01:55 aiaustin Note Added: 0035381
2019-06-11 01:57 aiaustin Note Edited: 0035381 View Revisions
2019-06-11 01:57 aiaustin Note Edited: 0035381 View Revisions
2019-06-11 02:00 aiaustin Note Edited: 0035381 View Revisions
2019-06-11 22:31 UbitUmarov Note Added: 0035392
2019-06-12 00:32 aiaustin Note Edited: 0035392 View Revisions
2019-07-31 01:45 mewtwo0641 Note Added: 0035517
2019-08-20 06:46 mike.dickson Note Added: 0035595
2019-08-23 06:48 mike.dickson Note Added: 0035596
2019-08-23 12:06 UbitUmarov Note Added: 0035597
2019-08-23 12:10 UbitUmarov Note Added: 0035598
2019-08-23 13:21 aiaustin Note Added: 0035599
2019-08-23 13:25 aiaustin Note Edited: 0035599 View Revisions
2019-08-24 02:17 UbitUmarov Note Added: 0035601
2019-08-24 04:08 aiaustin Note Added: 0035602
2019-08-24 04:08 aiaustin Note Edited: 0035598 View Revisions
2019-08-24 04:11 aiaustin Note Edited: 0035602 View Revisions
2019-08-24 04:14 aiaustin Note Edited: 0035602 View Revisions
2019-08-24 04:20 aiaustin Note Edited: 0035602 View Revisions
2019-08-24 08:06 aiaustin Note Added: 0035603
2019-08-24 08:09 aiaustin Note Edited: 0035602 View Revisions
2019-08-24 08:10 aiaustin Note Edited: 0035602 View Revisions
2019-08-24 13:36 UbitUmarov Note Added: 0035604
2019-08-24 13:43 Total Sorbet Note Added: 0035605
2019-08-24 13:44 Total Sorbet Note Edited: 0035605 View Revisions
2019-08-24 13:44 Total Sorbet Note Edited: 0035605 View Revisions
2019-08-24 13:52 Total Sorbet Note Edited: 0035605 View Revisions
2019-08-24 15:04 UbitUmarov Note Added: 0035606
2019-08-24 15:58 UbitUmarov Note Added: 0035607
2019-08-24 16:00 UbitUmarov Note Edited: 0035607 View Revisions
2019-08-24 16:04 UbitUmarov Note Edited: 0035607 View Revisions
2019-08-24 16:37 UbitUmarov Note Added: 0035608
2019-08-24 17:34 mike.dickson Note Added: 0035609
2019-08-25 09:56 aiaustin Note Added: 0035610
2019-08-27 03:53 aiaustin Note Added: 0035612
2019-08-27 03:54 aiaustin Note Edited: 0035612 View Revisions
2019-08-27 03:57 aiaustin Note Edited: 0035612 View Revisions
2019-08-27 10:53 UbitUmarov Note Added: 0035614
2019-08-28 08:38 Total Sorbet Note Added: 0035622
2019-08-31 05:04 mike.dickson Note Added: 0035625
2019-08-31 06:37 UbitUmarov Note Added: 0035626
2019-09-01 05:43 mike.dickson Note Added: 0035631
2019-09-01 06:58 UbitUmarov Note Added: 0035633
2019-09-02 18:52 mewtwo0641 Note Added: 0035656
2019-09-03 02:29 aiaustin Note Added: 0035658
2019-09-03 02:30 aiaustin Note Edited: 0035658 View Revisions
2019-09-07 18:32 mewtwo0641 Note Added: 0035676
2019-09-07 18:32 mewtwo0641 Note Edited: 0035676 View Revisions
2019-12-03 10:33 gimisa Note Added: 0035922
2019-12-03 11:06 Ferd Frederix Note Added: 0035925
2019-12-03 11:08 UbitUmarov Note Added: 0035926
2019-12-03 11:18 aiaustin Note Added: 0035927
2019-12-03 11:19 aiaustin Note Edited: 0035927 View Revisions
2019-12-03 12:45 gimisa Note Added: 0035929
2019-12-03 12:47 UbitUmarov Note Added: 0035930
2019-12-03 14:19 aiaustin Note Edited: 0035927 View Revisions
2019-12-03 14:20 aiaustin Note Edited: 0035927 View Revisions
2019-12-03 23:04 tampa Note Added: 0035934
2019-12-04 04:38 gimisa Note Added: 0035935
2019-12-04 04:43 gimisa Note Edited: 0035935 View Revisions
2019-12-04 04:54 UbitUmarov Note Added: 0035936
2019-12-04 12:02 gimisa Note Added: 0035937
2019-12-04 12:03 gimisa Note Edited: 0035937 View Revisions
2019-12-04 12:08 UbitUmarov Note Added: 0035938
2019-12-05 03:27 gimisa Note Added: 0035940
2019-12-05 03:30 gimisa Note Edited: 0035940 View Revisions
2019-12-05 05:40 UbitUmarov Note Added: 0035941
2019-12-06 15:49 alanscotch@yahoo.com Note Added: 0035945
2019-12-06 18:27 gimisa Note Added: 0035946
2019-12-07 04:42 gimisa Note Added: 0035947
2019-12-07 04:47 gimisa Note Edited: 0035947 View Revisions
2019-12-07 10:10 alanscotch@yahoo.com Note Edited: 0035945 View Revisions
2019-12-07 11:02 alanscotch@yahoo.com Note Added: 0035948
2019-12-08 08:54 alanscotch@yahoo.com Note Added: 0035949
2019-12-08 08:58 UbitUmarov Note Added: 0035950
2019-12-08 09:01 UbitUmarov Note Added: 0035951
2019-12-08 12:55 alanscotch@yahoo.com Note Edited: 0035949 View Revisions
2019-12-08 12:56 alanscotch@yahoo.com Note Added: 0035952
2019-12-08 13:34 alanscotch@yahoo.com Note Edited: 0035952 View Revisions
2019-12-08 15:33 alanscotch@yahoo.com Note Edited: 0035952 View Revisions
2019-12-08 20:13 gimisa Note Added: 0035953
2019-12-09 03:03 UbitUmarov Note Added: 0035954
2019-12-09 07:31 aiaustin Note Added: 0035955
2019-12-09 07:31 aiaustin Note Edited: 0035955 View Revisions
2019-12-09 07:32 aiaustin Note Edited: 0035955 View Revisions
2019-12-09 07:33 aiaustin Note Edited: 0035955 View Revisions
2019-12-09 07:48 aiaustin Note Edited: 0035955 View Revisions
2019-12-09 09:04 gimisa Note Added: 0035956
2019-12-09 09:06 alanscotch@yahoo.com Note Added: 0035957
2019-12-09 09:08 gimisa Note Edited: 0035956 View Revisions
2019-12-10 04:24 UbitUmarov Note Added: 0035958
2019-12-10 05:11 gimisa Note Added: 0035959
2019-12-10 05:16 UbitUmarov Note Added: 0035960
2019-12-10 11:23 aiaustin Note Edited: 0035960 View Revisions
2019-12-10 11:29 alanscotch@yahoo.com Note Added: 0035961
2019-12-10 11:35 UbitUmarov Note Added: 0035962
2019-12-10 11:50 aiaustin Note Edited: 0035959 View Revisions
2019-12-10 17:22 alanscotch@yahoo.com Note Edited: 0035961 View Revisions
2019-12-11 03:52 gimisa Note Added: 0035964
2019-12-11 05:20 UbitUmarov Note Added: 0035965
2019-12-11 06:32 aiaustin Note Added: 0035966
2019-12-11 06:33 aiaustin Note Edited: 0035966 View Revisions
2019-12-11 06:34 aiaustin Note Edited: 0035965 View Revisions
2019-12-11 06:35 aiaustin Note Edited: 0035965 View Revisions
2019-12-11 06:43 tampa Note Added: 0035969
2019-12-11 07:53 gimisa Note Added: 0035970
2019-12-11 08:00 aiaustin Note Added: 0035971
2019-12-11 08:00 aiaustin Note Edited: 0035971 View Revisions
2019-12-11 08:07 aiaustin Note Added: 0035972
2019-12-11 08:07 aiaustin Note Edited: 0035972 View Revisions
2019-12-11 08:18 aiaustin Note Edited: 0035972 View Revisions
2019-12-11 08:19 aiaustin Note Edited: 0035972 View Revisions
2019-12-11 08:22 aiaustin Note Added: 0035973
2019-12-11 08:29 aiaustin Note Edited: 0035973 View Revisions
2019-12-11 09:24 tampa Note Added: 0035974
2019-12-11 10:54 beqjanus Note Added: 0035975
2019-12-11 13:40 gimisa Note Added: 0035976
2019-12-12 01:50 aiaustin Note Added: 0035978
2019-12-12 01:50 aiaustin Note Edited: 0035976 View Revisions
2019-12-12 01:51 aiaustin Note Edited: 0035978 View Revisions
2019-12-12 09:03 gimisa Note Added: 0035980
2019-12-12 11:26 alanscotch@yahoo.com Note Added: 0035981
2019-12-12 14:44 aiaustin Note Added: 0035986
2019-12-12 14:45 aiaustin Note Edited: 0035986 View Revisions
2019-12-13 01:15 aiaustin Note Edited: 0035986 View Revisions
2019-12-13 09:38 UbitUmarov Note Added: 0035988
2019-12-13 12:44 gimisa Note Added: 0035989
2019-12-14 06:08 gimisa Note Edited: 0035989 View Revisions
2019-12-14 12:35 alanscotch@yahoo.com Note Added: 0035990
2019-12-14 12:54 alanscotch@yahoo.com Note Edited: 0035990 View Revisions
2019-12-14 13:31 gimisa Note Added: 0035991
2019-12-14 14:33 aiaustin Note Added: 0035992
2019-12-14 14:34 aiaustin Note Edited: 0035992 View Revisions
2019-12-14 14:56 tampa Note Added: 0035993
2019-12-14 15:23 gimisa Note Added: 0035994
2019-12-14 15:27 gimisa Note Edited: 0035994 View Revisions
2019-12-14 19:55 alanscotch@yahoo.com Note Edited: 0035990 View Revisions
2019-12-15 01:11 aiaustin Note Added: 0035995
2019-12-15 01:21 aiaustin Note Added: 0035996
2019-12-15 01:22 aiaustin Note Edited: 0035996 View Revisions
2019-12-19 03:14 UbitUmarov Note Added: 0035999
2019-12-19 06:06 aiaustin Note Added: 0036000
2019-12-19 07:55 gimisa Note Added: 0036001
2019-12-19 08:49 aiaustin Note Added: 0036002
2019-12-19 09:02 aiaustin Note Added: 0036003
2019-12-19 09:11 aiaustin Note Added: 0036004
2019-12-19 09:19 aiaustin Note Edited: 0036004 View Revisions
2019-12-19 09:19 UbitUmarov Note Added: 0036005
2019-12-19 09:19 aiaustin Note Edited: 0036003 View Revisions
2019-12-19 09:22 aiaustin Note Edited: 0036003 View Revisions
2019-12-19 09:26 aiaustin Note Edited: 0036002 View Revisions
2019-12-19 09:30 aiaustin Note Edited: 0036002 View Revisions
2019-12-19 09:30 aiaustin Note Edited: 0036004 View Revisions
2019-12-19 09:31 aiaustin Note Edited: 0036002 View Revisions
2019-12-19 09:31 aiaustin Note Edited: 0036002 View Revisions
2019-12-19 09:32 aiaustin Note Edited: 0036002 View Revisions
2019-12-19 09:33 aiaustin Note Edited: 0036003 View Revisions
2019-12-19 09:44 aiaustin Note Edited: 0036004 View Revisions
2019-12-19 11:15 aiaustin Note Edited: 0036002 View Revisions
2019-12-19 11:36 aiaustin Note Edited: 0036004 View Revisions
2019-12-19 11:37 aiaustin Note Edited: 0036003 View Revisions
2019-12-19 11:37 aiaustin Note Edited: 0036002 View Revisions
2019-12-19 12:04 aiaustin Note Edited: 0036003 View Revisions
2019-12-19 12:04 aiaustin Note Edited: 0036004 View Revisions
2019-12-19 12:05 aiaustin Note Edited: 0036003 View Revisions
2019-12-19 12:05 aiaustin Note Edited: 0036004 View Revisions
2019-12-19 12:05 aiaustin Note Edited: 0036002 View Revisions
2019-12-19 12:06 aiaustin Note Edited: 0036003 View Revisions
2019-12-19 13:45 aiaustin Note Added: 0036006
2019-12-19 13:55 aiaustin Note Edited: 0036006 View Revisions
2019-12-19 13:56 aiaustin Note Edited: 0036006 View Revisions
2019-12-19 13:56 aiaustin Note Edited: 0036002 View Revisions
2019-12-20 13:04 aiaustin Note Edited: 0036006 View Revisions
2019-12-20 13:05 aiaustin Note Edited: 0036006 View Revisions
2019-12-24 06:20 gimisa Note Added: 0036007
2019-12-24 06:55 aiaustin Note Added: 0036008
2019-12-24 10:39 gimisa Note Added: 0036009
2019-12-24 10:51 BillBlight Note Added: 0036010
2019-12-24 12:54 aiaustin Note Added: 0036011
2019-12-25 02:40 aiaustin Note Edited: 0036004 View Revisions
2019-12-26 01:49 aiaustin Note Edited: 0036011 View Revisions
2019-12-26 01:50 aiaustin Note Edited: 0036011 View Revisions
2019-12-26 06:00 mike.dickson Note Added: 0036012
2019-12-26 12:09 gimisa Note Added: 0036013
2019-12-30 10:08 aiaustin Note Added: 0036019
2019-12-30 10:09 aiaustin Note Edited: 0036019 View Revisions
2019-12-30 10:10 aiaustin Note Edited: 0036019 View Revisions
2019-12-30 11:03 aiaustin Note Edited: 0036019 View Revisions
2019-12-31 06:19 aiaustin Note Edited: 0036019 View Revisions
2019-12-31 14:58 aiaustin Note Added: 0036022
2019-12-31 15:02 mike.dickson Note Added: 0036023
2019-12-31 15:09 aiaustin Note Edited: 0036022 View Revisions
2019-12-31 15:14 aiaustin Note Added: 0036024
2020-01-01 06:30 aiaustin Note Edited: 0036022 View Revisions
2020-01-02 04:07 gimisa Note Added: 0036025


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker