Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008516opensim[REGION] OpenSim Corepublic2019-04-09 03:312019-06-11 22:31
Assigned ToUbitUmarov 
PlatformPCOSWindowsOS Version10
Product Versionmaster (dev code) 
Target Versionmaster (dev code)Fixed in Version 
Summary0008516: Missing Objects and/or Terrain Parts on Teleport between Grids
DescriptionThis is a Mantis issue to act as a place to note observations as the recent code changes related to SupportViewerObjectsCache are tried out more.

With recent dev master code on teleport there can be loss of objects and/or terrain chunks. They cannot be made visible by clicking in the area where the lost objects or terrain is known to be (which can occur already with some objects in viewers). So we are not able to get them back without moving away and back or relogging.

Sending a "force update" from the Server console also has no effect.

The recent commits suggest turning down the viewer bandwidth if this sort of loss occurs. I turned the viewer bandwidth down to 750KBps and that did seem to improve things a lot... but I still do see some items go AWOL occasionally even at 750Kbps. I have yet to identify the specifics of such a situation as its not all the time now.

I have also seen a couple of viewer crashes on teleport even at 750KBps bandwidth but again cannot be specific yet.

I do note a blip in packet loss in the stats tool (ctrl+shift+1) when the issues arises... Packet loss is on zero but as the loss occurs there is a sudden spoke a few seconds after arriving on a region where object/terrain loss can be observed.
Steps To ReproduceTeleport back and forth between grids running latest dev master code and observe if any objects or terrain patches are missing. Try viewer bandwidths such as 1500KBps (default for Firestorm on my setup) and 750KBps.
Additional InformationUbitUmarov and myself have discussed this quite a bit and done some cross grid hypergrid testing with a range of viewer bandwidths rather than my usual (the default I think) 1500Kbps I use on Firestorm (on a 100Mbps link at home and gigabit networking at work).

This occurred with both with SupportViewerObjectsCache set to false and true (true now being the default in master code). Ubit indicated the mechanism changed even if the parameter was set to false.

Ubit indicates that we should be resilient to reasonable packet loss. I have seen moderate packet loss on Second Life and on earlier OSGrid releases without any problems of such object loss. But here the visual loss of objects does seem to be associated with an observation in the stats console of a sudden spoke in packet loss as it occurs.
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (Multiple Regions per Sim)
Physics EngineBulletSim
Environment.NET / Windows64
Mono VersionNone
Viewerdev master (2019-04-09)
Attached Filesjpg file icon 2019-04-09-Terrain-Loss-1.jpg [^] (89,528 bytes) 2019-04-09 08:10

jpg file icon 2019-04-09-Prim-Loss-1.jpg [^] (182,032 bytes) 2019-04-09 08:22

- Relationships
related to 0008515resolvedUbitUmarov Mesh disappears 
related to 0008528assignedUbitUmarov Random mesh pieces do not rez 

-  Notes
aiaustin (developer)
2019-04-09 08:07

The way the stats for ping loss show it is as if the cause of the packet loss is occurring just after the arrival of the avatar at the destination region. Not that packet loss that just happens to occur causes the loss if the objects or terrain.

Packet loss can be registering zero, and seems to stay at zero whenever a successful TP takes place and no object or terrain is lost. But I think it is the case that each time I see packet loss if I look round I find there are missing prims or missing chunks of the terrain that are around 16m X 16m.
aiaustin (developer)
2019-04-09 08:21
edited on: 2019-04-09 08:23

Now I am observing something I find weird... note I am TPing between AILand grid Ailand region (avatar logged in natively there) and Openvue grid Openvue region (across Hypergrid). These are the two main regions on each grid. I am on latest dev master on each (2019-04-09 12:05). I am going slow on TP rate, leaving 20 seconds between TPs.

Once I get the prims missing (in this case three ground level 10x10 prims in the cobblestone area on Openvue as shown in attached image 2019-04-09-Prim-Loss-1.jpg), I can TP back home and return to Openvue and the SAME prims are missing. I can even relog, do the same and they can be missing.

BUT, and this is the odd thing... but might hold a clue to where the prims are getting lost... if when on the foreign grid Openvue region I TP via the map to an adjacent region (Sandbox) and the n return via the map to Openvue all on the same grid, the prims show.

BUT, then I go home to AiLand and TP back to Openvue again (all leaving 20 seconds between jumps) and the SAME prims have gone again. Its as if a totally different mechanism is being used to refresh the contents for inter-grid to inter-region jumps. I told you it was weird.

UbitUmarov (administrator)
2019-04-09 08:45

been there via HG with 1500kbps fs and had no issues
aiaustin (developer)
2019-04-09 08:47
edited on: 2019-04-09 08:53

Good :-) I am watching the logs as you hop about Ubit.

All looks normal. This showed once...

16:53:03 - [LLUDPSERVER]: Received a resend of already processed packet 0000047, type RegionHandshakeReply from Ubit.Umarov

mewtwo0641 (reporter)
2019-04-09 22:03

I see this issue still on master with grid set up as not HG. Have to constantly issue 'force update' command in the console to get the missing pieces to show.
aiaustin (developer)
2019-04-10 02:04
edited on: 2019-04-10 02:13

I updated to (2019-04-09 23:27) for both grids and observe essentially the same issues. At 750Kbps it works most of the time, but at 1500Kbps it fails most of the time. And after I have prims and terrain missing even if I wait 20 seconds between a TP out and back the same bits are still missing.

As before a small packet loss just once was associated with the first appearance of the missing prims/terrain chunks and after that it sticks even with nno further reported pin loss. It looks like once lost its.

BUT.. and I still find this peculiar and perhaps a hint as to what might be happening... a TP to an adjacent region (Sandbox) on the second grid and back to the region on that grid with missing prims/terrain (Openvue) makes everything show fine, but TP back to home grid (AiLand) and back to Openvue and the same bits are missing again.

I have double checked, and "force update" has no effect on the missing prims or terrain chunks for me on Openvue/AiLand grid testing. I think MeMewToo0641 may have a separate issue as in all my testing I have never been able to make any prim or terrain chunk that goes missing reappear.

aiaustin (developer)
2019-04-10 02:16
edited on: 2019-04-10 02:19

Ohoh… now the missing chunks and prims are persisting across relogs. I logged out and back in and after all settled did a jump to Openvue grid and the same bits are missing there.

In two separate relogs, I logged in to AiLand and let everything settle down. then I did my first jump to Openvue grid.. and the parts missing on previous logins were still missing. Same parts.

aiaustin (developer)
2019-04-10 09:09
edited on: 2019-04-10 09:10

I still have the same prims missing on FIRST teleport from AiLand grid to Openvue grid after updating from yesterday's git master (966 - 2019-04-09 23:27) to today's (967 - 2019-04-10 13:01) and hence restarting the servers.

Is this possible? I thought the object cache got refreshed after server restart?

UbitUmarov (administrator)
2019-04-10 09:19
edited on: 2019-04-11 09:41

no direct relation to objects cache.
as I told several times, viewer just does not rez things it did received.
most likely because it did overload, leaving things in memory in broken state.
the effect ranges from totally missing, just invisible that show up on touch and even just a face or two visible.
What prims, depends just on order they did arrive to viewer and how the issue was triggered.

similar for terrain.

viewer bandwidth is not only about network, is also about viewer capability to process things.

aiaustin (developer)
2019-04-10 10:03

If it is in memory Ubit, why does it persist over relogs/viewer restarts, and even after server clean upgrades and restarts?

Also, how come the objects show when moving off to an adjacent region and back on the destination grid, yet the same missing parts are still missing after return to home grid and back. Is there a different mechanism for updates in these two cases?

Clearly something serious has gone wrong.. we have not seen issues like this before have we? Its a very different issue to the occasional non-rez of objects that show when the area they are in is clicked on.
BillBlight (developer)
2019-04-10 10:19

@aiaustin, the same could be said, for WHY am I and others on my grid, on osgrid, and on 2 test grids unable to replicate your issue ..

Stomping your foot and saying it's broken does not help the issue.

I can give you 4 cases where it is not broken.

Not trying to say it is not an issue somewhere but two people having an issue does not point to a systemic issue.
aiaustin (developer)
2019-04-10 11:03

Bill.. I am not “stomping my foot”. I am reporting a clear change in behaviour on my test grids. I am delighted that you don’t see any issue on your specific test environments. This Mantis Issue is here to collect information as and when or if anyone else starts to see the issues, so we can pin down the problem.
aiaustin (developer)
2019-04-13 01:00
edited on: 2019-04-13 01:01

Updated Openvue and AiLand grids to (2019-04-11 23:19) and set SupportViewerObjectsCache = false explicitly in OpenSim.ini (default now being true from openSimDefaults.ini).

Testing over the last 2 days in two viewer environments one of which is on a fast link on the same subnet as the servers and the other is over a home ISP accessing the servers remotely. using Firestorm (Windows 10 Pro), default bandwidth of 1500Kbps.

I do not see any missing prims or terrain chunks on teleports between these grids with SupportViewerObjectsCache = false. I even tested TPs back and forth 5 or 6 times at under 5 seconds delay. All looks fine. No packet loss blip occurs when using the stats console.

I then restarted the two grids with SupportViewerObjectsCache = true on both and then see a few missing prims when doing TPs. The prims this time differed a little on each TP, but most were apparently sub-parts of a linked object.

I did these tests after seeing the commit (2019-04-10 20:28) which may have fixed the problem with the issue occurring even when SupportViewerObjectsCache = false as reported earlier.

BillBlight (developer)
2019-04-13 06:25

Just for information purposes, FS default bandwidth is 500Kbps not 1500.
aiaustin (developer)
2019-04-13 09:57
edited on: 2019-04-13 10:08

Ah, that’s good to know Bill. I was just concerned to make sure we work well with defaults, as many users will never change such settings. I guess I set mine to “cable” speed a long time ago and did not realise that continued to be set.

aiaustin (developer)
2019-06-05 11:19

This issue now seems to be fixed.

Openvue and AiLand grids now use SupportViewerObjectsCache = true (the default as set in openSimDefaults.ini).

Issue is addressed on latest dev master - sometime between 24th March 2019 and 14th May 2019. So OSgrid version as of 14th May 2019 is good too.
aiaustin (developer)
2019-06-11 01:55
edited on: 2019-06-11 02:00

As at OSGrid 2019-05-14 I do see missing prims at initial login. Not TP or HG related. Simple cubes missing. Different ones in same linkset for example none after several logins. As before, right clicking or trying to operate on the affected object does not make the missing piece rez. A relog is needed.

UbitUmarov (administrator)
2019-06-11 22:31
edited on: 2019-06-12 00:32

let me remark that viewer capability to handle prims may depend on fps
try to keep the viewer maximized and selected during all login/tps

- Issue History
Date Modified Username Field Change
2019-04-09 03:31 aiaustin New Issue
2019-04-09 03:31 aiaustin Status new => assigned
2019-04-09 03:31 aiaustin Assigned To => UbitUmarov
2019-04-09 03:33 aiaustin Relationship added related to 0008515
2019-04-09 03:35 aiaustin Additional Information Updated View Revisions
2019-04-09 08:07 aiaustin Note Added: 0035073
2019-04-09 08:09 aiaustin File Added: 2019-04-09-Terrain-Loss-1.jpg
2019-04-09 08:09 aiaustin File Deleted: 2019-04-09-Terrain-Loss-1.jpg
2019-04-09 08:10 aiaustin File Added: 2019-04-09-Terrain-Loss-1.jpg
2019-04-09 08:21 aiaustin Note Added: 0035074
2019-04-09 08:22 aiaustin File Added: 2019-04-09-Prim-Loss-1.jpg
2019-04-09 08:23 aiaustin Note Edited: 0035074 View Revisions
2019-04-09 08:45 UbitUmarov Note Added: 0035075
2019-04-09 08:47 aiaustin Note Added: 0035076
2019-04-09 08:53 aiaustin Note Edited: 0035076 View Revisions
2019-04-09 22:03 mewtwo0641 Note Added: 0035077
2019-04-10 02:04 aiaustin Note Added: 0035078
2019-04-10 02:04 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:08 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:10 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:12 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:13 aiaustin Note Edited: 0035078 View Revisions
2019-04-10 02:16 aiaustin Note Added: 0035079
2019-04-10 02:19 aiaustin Note Edited: 0035079 View Revisions
2019-04-10 09:09 aiaustin Note Added: 0035080
2019-04-10 09:10 aiaustin Note Edited: 0035080 View Revisions
2019-04-10 09:19 UbitUmarov Note Added: 0035081
2019-04-10 10:03 aiaustin Note Added: 0035082
2019-04-10 10:19 BillBlight Note Added: 0035083
2019-04-10 11:03 aiaustin Note Added: 0035084
2019-04-11 09:41 aiaustin Note Edited: 0035081 View Revisions
2019-04-13 01:00 aiaustin Note Added: 0035085
2019-04-13 01:01 aiaustin Note Edited: 0035085 View Revisions
2019-04-13 06:25 BillBlight Note Added: 0035086
2019-04-13 09:57 aiaustin Note Added: 0035092
2019-04-13 10:07 aiaustin Note Edited: 0035092 View Revisions
2019-04-13 10:08 aiaustin Note Edited: 0035092 View Revisions
2019-05-14 09:06 aiaustin Relationship added related to 0008528
2019-06-05 11:19 aiaustin Note Added: 0035288
2019-06-11 01:55 aiaustin Note Added: 0035381
2019-06-11 01:57 aiaustin Note Edited: 0035381 View Revisions
2019-06-11 01:57 aiaustin Note Edited: 0035381 View Revisions
2019-06-11 02:00 aiaustin Note Edited: 0035381 View Revisions
2019-06-11 22:31 UbitUmarov Note Added: 0035392
2019-06-12 00:32 aiaustin Note Edited: 0035392 View Revisions

Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker