|Anonymous | Login | Signup for a new account||2019-05-27 03:32 PDT|
|Main | My View | View Issues | Change Log | Roadmap | Summary | My Account|
|View Issue Details|
|ID||Project||Category||View Status||Date Submitted||Last Update|
|0008516||opensim||[REGION] OpenSim Core||public||2019-04-09 03:31||2019-05-14 09:06|
|Product Version||master (dev code)|
|Target Version||master (dev code)||Fixed in Version|
|Summary||0008516: Missing Objects and/or Terrain Parts on Teleport between Grids|
|Description||This is a Mantis issue to act as a place to note observations as the recent code changes related to SupportViewerObjectsCache are tried out more.|
With recent dev master code on teleport there can be loss of objects and/or terrain chunks. They cannot be made visible by clicking in the area where the lost objects or terrain is known to be (which can occur already with some objects in viewers). So we are not able to get them back without moving away and back or relogging.
Sending a "force update" from the Server console also has no effect.
The recent commits suggest turning down the viewer bandwidth if this sort of loss occurs. I turned the viewer bandwidth down to 750KBps and that did seem to improve things a lot... but I still do see some items go AWOL occasionally even at 750Kbps. I have yet to identify the specifics of such a situation as its not all the time now.
I have also seen a couple of viewer crashes on teleport even at 750KBps bandwidth but again cannot be specific yet.
I do note a blip in packet loss in the stats tool (ctrl+shift+1) when the issues arises... Packet loss is on zero but as the loss occurs there is a sudden spoke a few seconds after arriving on a region where object/terrain loss can be observed.
|Steps To Reproduce||Teleport back and forth between grids running latest dev master code and observe if any objects or terrain patches are missing. Try viewer bandwidths such as 1500KBps (default for Firestorm on my setup) and 750KBps.|
|Additional Information||UbitUmarov and myself have discussed this quite a bit and done some cross grid hypergrid testing with a range of viewer bandwidths rather than my usual (the default I think) 1500Kbps I use on Firestorm (on a 100Mbps link at home and gigabit networking at work). |
This occurred with both with SupportViewerObjectsCache set to false and true (true now being the default in master code). Ubit indicated the mechanism changed even if the parameter was set to false.
Ubit indicates that we should be resilient to reasonable packet loss. I have seen moderate packet loss on Second Life and on earlier OSGrid releases without any problems of such object loss. But here the visual loss of objects does seem to be associated with an observation in the stats console of a sudden spoke in packet loss as it occurs.
|Tags||No tags attached.|
|Git Revision or version number|
|Run Mode||Grid (Multiple Regions per Sim)|
|Environment||.NET / Windows64|
|Viewer||dev master (2019-04-09)|
|Attached Files|| 2019-04-09-Terrain-Loss-1.jpg [^] (89,528 bytes) 2019-04-09 08:10
2019-04-09-Prim-Loss-1.jpg [^] (182,032 bytes) 2019-04-09 08:22
The way the stats for ping loss show it is as if the cause of the packet loss is occurring just after the arrival of the avatar at the destination region. Not that packet loss that just happens to occur causes the loss if the objects or terrain.
Packet loss can be registering zero, and seems to stay at zero whenever a successful TP takes place and no object or terrain is lost. But I think it is the case that each time I see packet loss if I look round I find there are missing prims or missing chunks of the terrain that are around 16m X 16m.
edited on: 2019-04-09 08:23
Now I am observing something I find weird... note I am TPing between AILand grid Ailand region (avatar logged in natively there) and Openvue grid Openvue region (across Hypergrid). These are the two main regions on each grid. I am on latest dev master on each opensim-0.9.0.1-962-g49fb9d6.zip (2019-04-09 12:05). I am going slow on TP rate, leaving 20 seconds between TPs.
Once I get the prims missing (in this case three ground level 10x10 prims in the cobblestone area on Openvue as shown in attached image 2019-04-09-Prim-Loss-1.jpg), I can TP back home and return to Openvue and the SAME prims are missing. I can even relog, do the same and they can be missing.
BUT, and this is the odd thing... but might hold a clue to where the prims are getting lost... if when on the foreign grid Openvue region I TP via the map to an adjacent region (Sandbox) and the n return via the map to Openvue all on the same grid, the prims show.
BUT, then I go home to AiLand and TP back to Openvue again (all leaving 20 seconds between jumps) and the SAME prims have gone again. Its as if a totally different mechanism is being used to refresh the contents for inter-grid to inter-region jumps. I told you it was weird.
|been there via HG with 1500kbps fs and had no issues|
edited on: 2019-04-09 08:53
Good :-) I am watching the logs as you hop about Ubit.
All looks normal. This showed once...
16:53:03 - [LLUDPSERVER]: Received a resend of already processed packet 0000047, type RegionHandshakeReply from Ubit.Umarov @hg.osgrid.org
|I see this issue still on master with grid set up as not HG. Have to constantly issue 'force update' command in the console to get the missing pieces to show.|
edited on: 2019-04-10 02:13
I updated to opensim-0.9.0.1-966-gee989dd.zip (2019-04-09 23:27) for both grids and observe essentially the same issues. At 750Kbps it works most of the time, but at 1500Kbps it fails most of the time. And after I have prims and terrain missing even if I wait 20 seconds between a TP out and back the same bits are still missing.
As before a small packet loss just once was associated with the first appearance of the missing prims/terrain chunks and after that it sticks even with nno further reported pin loss. It looks like once lost its.
BUT.. and I still find this peculiar and perhaps a hint as to what might be happening... a TP to an adjacent region (Sandbox) on the second grid and back to the region on that grid with missing prims/terrain (Openvue) makes everything show fine, but TP back to home grid (AiLand) and back to Openvue and the same bits are missing again.
I have double checked, and "force update" has no effect on the missing prims or terrain chunks for me on Openvue/AiLand grid testing. I think MeMewToo0641 may have a separate issue as in all my testing I have never been able to make any prim or terrain chunk that goes missing reappear.
edited on: 2019-04-10 02:19
Ohoh… now the missing chunks and prims are persisting across relogs. I logged out and back in and after all settled did a jump to Openvue grid and the same bits are missing there.
In two separate relogs, I logged in to AiLand and let everything settle down. then I did my first jump to Openvue grid.. and the parts missing on previous logins were still missing. Same parts.
edited on: 2019-04-10 09:10
I still have the same prims missing on FIRST teleport from AiLand grid to Openvue grid after updating from yesterday's git master (966 - 2019-04-09 23:27) to today's (967 - 2019-04-10 13:01) and hence restarting the servers.
Is this possible? I thought the object cache got refreshed after server restart?
edited on: 2019-04-11 09:41
no direct relation to objects cache.
as I told several times, viewer just does not rez things it did received.
most likely because it did overload, leaving things in memory in broken state.
the effect ranges from totally missing, just invisible that show up on touch and even just a face or two visible.
What prims, depends just on order they did arrive to viewer and how the issue was triggered.
similar for terrain.
viewer bandwidth is not only about network, is also about viewer capability to process things.
If it is in memory Ubit, why does it persist over relogs/viewer restarts, and even after server clean upgrades and restarts?
Also, how come the objects show when moving off to an adjacent region and back on the destination grid, yet the same missing parts are still missing after return to home grid and back. Is there a different mechanism for updates in these two cases?
Clearly something serious has gone wrong.. we have not seen issues like this before have we? Its a very different issue to the occasional non-rez of objects that show when the area they are in is clicked on.
@aiaustin, the same could be said, for WHY am I and others on my grid, on osgrid, and on 2 test grids unable to replicate your issue ..
Stomping your foot and saying it's broken does not help the issue.
I can give you 4 cases where it is not broken.
Not trying to say it is not an issue somewhere but two people having an issue does not point to a systemic issue.
|Bill.. I am not “stomping my foot”. I am reporting a clear change in behaviour on my test grids. I am delighted that you don’t see any issue on your specific test environments. This Mantis Issue is here to collect information as and when or if anyone else starts to see the issues, so we can pin down the problem.|
edited on: 2019-04-13 01:01
Updated Openvue and AiLand grids to opensim-0.9.0.1-971-gcfd3923.zip (2019-04-11 23:19) and set SupportViewerObjectsCache = false explicitly in OpenSim.ini (default now being true from openSimDefaults.ini).
Testing over the last 2 days in two viewer environments one of which is on a fast link on the same subnet as the servers and the other is over a home ISP accessing the servers remotely. using Firestorm 188.8.131.52680 (Windows 10 Pro), default bandwidth of 1500Kbps.
I do not see any missing prims or terrain chunks on teleports between these grids with SupportViewerObjectsCache = false. I even tested TPs back and forth 5 or 6 times at under 5 seconds delay. All looks fine. No packet loss blip occurs when using the stats console.
I then restarted the two grids with SupportViewerObjectsCache = true on both and then see a few missing prims when doing TPs. The prims this time differed a little on each TP, but most were apparently sub-parts of a linked object.
I did these tests after seeing the commit opensim-0.9.0.1-968-gd977613.zip (2019-04-10 20:28) which may have fixed the problem with the issue occurring even when SupportViewerObjectsCache = false as reported earlier.
|Just for information purposes, FS default bandwidth is 500Kbps not 1500.|
edited on: 2019-04-13 10:08
Ah, that’s good to know Bill. I was just concerned to make sure we work well with defaults, as many users will never change such settings. I guess I set mine to “cable” speed a long time ago and did not realise that continued to be set.
|2019-04-09 03:31||aiaustin||New Issue|
|2019-04-09 03:31||aiaustin||Status||new => assigned|
|2019-04-09 03:31||aiaustin||Assigned To||=> UbitUmarov|
|2019-04-09 03:33||aiaustin||Relationship added||related to 0008515|
|2019-04-09 03:35||aiaustin||Additional Information Updated||View Revisions|
|2019-04-09 08:07||aiaustin||Note Added: 0035073|
|2019-04-09 08:09||aiaustin||File Added: 2019-04-09-Terrain-Loss-1.jpg|
|2019-04-09 08:09||aiaustin||File Deleted: 2019-04-09-Terrain-Loss-1.jpg|
|2019-04-09 08:10||aiaustin||File Added: 2019-04-09-Terrain-Loss-1.jpg|
|2019-04-09 08:21||aiaustin||Note Added: 0035074|
|2019-04-09 08:22||aiaustin||File Added: 2019-04-09-Prim-Loss-1.jpg|
|2019-04-09 08:23||aiaustin||Note Edited: 0035074||View Revisions|
|2019-04-09 08:45||UbitUmarov||Note Added: 0035075|
|2019-04-09 08:47||aiaustin||Note Added: 0035076|
|2019-04-09 08:53||aiaustin||Note Edited: 0035076||View Revisions|
|2019-04-09 22:03||mewtwo0641||Note Added: 0035077|
|2019-04-10 02:04||aiaustin||Note Added: 0035078|
|2019-04-10 02:04||aiaustin||Note Edited: 0035078||View Revisions|
|2019-04-10 02:08||aiaustin||Note Edited: 0035078||View Revisions|
|2019-04-10 02:10||aiaustin||Note Edited: 0035078||View Revisions|
|2019-04-10 02:12||aiaustin||Note Edited: 0035078||View Revisions|
|2019-04-10 02:13||aiaustin||Note Edited: 0035078||View Revisions|
|2019-04-10 02:16||aiaustin||Note Added: 0035079|
|2019-04-10 02:19||aiaustin||Note Edited: 0035079||View Revisions|
|2019-04-10 09:09||aiaustin||Note Added: 0035080|
|2019-04-10 09:10||aiaustin||Note Edited: 0035080||View Revisions|
|2019-04-10 09:19||UbitUmarov||Note Added: 0035081|
|2019-04-10 10:03||aiaustin||Note Added: 0035082|
|2019-04-10 10:19||BillBlight||Note Added: 0035083|
|2019-04-10 11:03||aiaustin||Note Added: 0035084|
|2019-04-11 09:41||aiaustin||Note Edited: 0035081||View Revisions|
|2019-04-13 01:00||aiaustin||Note Added: 0035085|
|2019-04-13 01:01||aiaustin||Note Edited: 0035085||View Revisions|
|2019-04-13 06:25||BillBlight||Note Added: 0035086|
|2019-04-13 09:57||aiaustin||Note Added: 0035092|
|2019-04-13 10:07||aiaustin||Note Edited: 0035092||View Revisions|
|2019-04-13 10:08||aiaustin||Note Edited: 0035092||View Revisions|
|2019-05-14 09:06||aiaustin||Relationship added||related to 0008528|
|Copyright © 2000 - 2012 MantisBT Group|