Mantis Bug Tracker

View Issue Details Jump to Notes ] Issue History ] Print ]
IDProjectCategoryView StatusDate SubmittedLast Update
0008666opensim[REGION] OpenSim Corepublic2020-03-01 14:242020-03-17 02:31
ReporterKitto Flora 
Assigned To 
PrioritynormalSeverityminorReproducibilityrandom
StatusnewResolutionopen 
PlatformAMD FX(tm)-4170 Quad-coreOSLinux XubuntuOS Version16.04
Product Version0.9.1.0 
Target VersionFixed in Version 
Summary0008666: Client view jumps to old location for a few seconds, then back. Repeats.
DescriptionClient view will suddenly jump to where the avatar was some minutes ago, for a few seconds, then back to current spot. If no kb input view will continue to jump back and forth.
Another manifestation - while watching passing vehicle, view will suddenly jump to vehicle moving elsewhere for a few seconds, then back to current spot.

Re-log fixes the problem, simply moving the avatar may not.
Steps To ReproduceMove around a region for a while. Large region with a lot of content may help reproduce. May take more than 30 minutes to manifest.
TagsNo tags attached.
Git Revision or version number
Run Mode Grid (Multiple Regions per Sim)
Physics EngineubODE
Script EngineXEngine
EnvironmentMono / Linux64
Mono VersionNone
ViewerSingularity 8232 & FS
Attached Files

- Relationships

-  Notes
(0036236)
UbitUmarov (administrator)
2020-03-01 15:40

pls try with plain master code ( without NPC tweak)
(0036239)
mewtwo0641 (reporter)
2020-03-02 22:21

I am seeing this issue as well, on master. I'm not using any NPC tweaks that I know of.
(0036240)
UbitUmarov (administrator)
2020-03-02 22:50
edited on: 2020-03-02 22:51

well the description is not very clear, specially on the second case, that may be unrelated issue
can't tell, can't repo that
btw that is packet loss, and viewer bandwidth setting?

(0036241)
tampa (reporter)
2020-03-02 23:23

I am having some difficulty understanding exactly what you guys are seeing, there a possibility you could record the findings on video?
(0036243)
mewtwo0641 (reporter)
2020-03-03 02:11

It is difficult to reproduce on command (I haven't been successful at least) so getting a video would also be difficult.

The best way I can explain it is, I could be standing perfectly still not doing anything, and my av will just all of a sudden just float away. It's kind of the same effect you might see when trying to cross over into a new region and you see your avatar float/skate away for a little bit until the viewer establishes connection to the new region. If I press an arrow key to move, then the avatar will pop back to where it was originally standing, then a few seconds later it'll float away again.

I am pretty sure it isn't packet loss as the server instance I'm connecting to is on my LAN and both my viewer's computer and my OpenSim computer are hardwired to the router. Have only been seeing this issue very very recently.

Bandwidth set at 1000
(0036244)
Luisillo_Contepomi (reporter)
2020-03-03 02:20
edited on: 2020-03-03 02:21

I will try to explain it a little more.
I have not been able to reproduce it always. Sometimes it doesn't happen.

I arrive in a region and I am in the place for a minute. I go to another position within the same region with double click or using the map.

In the new position, I look at my avatar and wait a couple of minutes. Suddenly I can no longer see myself. My avatar has disappeared from my view and has returned to the previous position.

The last address appears correctly in the address bar but my avatar is not visible. My avatar is in the first position when I arrived in the region.
If I am sitting when I have disappeared .. when I get up the avatar appears again in the view

I have not reported it because I have been unable to reproduce it always.

Tested with FS, Dayturn and Cool VL Alpha (Last).

(0036245)
mewtwo0641 (reporter)
2020-03-03 03:21

I'm not sure if it's related but, I have been seeing this strange error in the console recently too at more or less the same time I see the floating happening

[UDPBASE]: sync send SocketException 0.0.0.0:0 The requested address is not valid in its context
(0036247)
Kitto Flora (reporter)
2020-03-04 12:47

Here is a movie of this problem happening
https://www.dropbox.com/s/qplur1fpkvieo5k/OSrepeat.mov?dl=0 [^]
At the start of the movie the Avatar is in the correct position.
Suddenly the Av moves to the SE and stops at edge of region. After a while
Av jumps back to correct position. This cycle repeats. During this time there is no user keyboard input.
At the bottom is the Client console. Client is Singularity 8232. At one point I scrolled the console up and down to make sure it is showing latest messages.
No console messages relevant to the Av motion are seen.
OS version is "OpenSim 0.9.2.0 Yeti DreamNation" and has the NPC physics fix.
(0036248)
tampa (reporter)
2020-03-04 13:01

That looks really odd, does this happen on master code as well or just their version?

Is there anything on the simulator console about resending packets?
(0036249)
UbitUmarov (administrator)
2020-03-04 13:27

Thanks for the extra work Kitto
so that move was from a server side autopilot from clicking on groud, or menu move to?
(0036250)
UbitUmarov (administrator)
2020-03-04 16:42

the float away mewtwo0641 mention maybe another issue. A physics collision decide to apply instante spurius velocity to the avatar, and a stop was never sent
viewer will just show it moving
(0036251)
Kitto Flora (reporter)
2020-03-04 17:06

tampa: The OS code versions I can easily change between are 0.9.0, 0.9.1 and 0.9.2+NPC fix. I have requested that 0.9.2 without the NPC fix be made available on the grid (by the grid management officer) but no response yet. You must be specific about what version you want me to try. Anything other than the above three will take a lot of time to make available on the grid.
I can (usually) call up the simulator console, given some time. If I see this effect again I will try access the console and look for such a message.
Ubit: This was not started by autopilot click or menu move to. I agree that is what it looks like, but for some time (15 secs +) before the repeat motion started the Av was standing still watching the train position tell-tail (yellow float text). The target location for a move-to was off-screen.
Also there is nothing physical anywhere near the Av, and no other Av in the region at this time.
(0036252)
UbitUmarov (administrator)
2020-03-04 17:18

looking to the video again, it also looks like a spurius velocity that the viewer interpolators pick and move the avatar..
no idea
tested and saw nothing :(
(0036254)
UbitUmarov (administrator)
2020-03-05 04:36

Luisillo, that is other issue. And should be fixed on master now
(0036255)
Luisillo_Contepomi (reporter)
2020-03-05 05:35

Thank you
(0036256)
tampa (reporter)
2020-03-05 10:46

It should be trivial to create a binary and setup a region with it, if not then I suspect there are more "fixes" applied to the binaries that may fail to merge. We need to be able to reproduce it somehow, without that it becomes guessing. Even if Ubit has an idea of what the cause may be, in the end to test the actual fix you would still need to build master code.

If the grid management struggles with that they can ask for help on IRC or consult the wiki, the instructions are on there.
(0036263)
Kitto Flora (reporter)
2020-03-07 19:23

More clues: We have set up a copy of the big (3x3) region where this problem has been observed before. It is running 0.9.2 + NPC fix. Only my Av is on the region. After being active for several hours, then idle (away) for at least an hour I discover that the Av is stuck in place - cannot move. Of the surrounding scenery there is part of a train moving along the track, then jumping back and repeating the motion - maybe 20 seconds to repeat this loop. On the simulator console there is a message repeating at about 2x per second:
[UDPBASE]: sync send SocketException 0.0.0.0:0 Invalid arguments
scrolling back some minutes (and many of above messages) there is:
[UDPBASE]: Error processing UDP end receive 926658. Exception System.ArgumentNullException: Value cannot be null.
Parameter name: endPoint
  at System.Net.Sockets.Socket.EndReceiveFrom (System.IAsyncResult asyncResult, System.Net.EndPoint& endPoint) [0x00049] in /home/kunta/mono-6.8.0-96-mmr/mcs/class/System/System.Net.Sockets/Socket.cs:1865
  at OpenMetaverse.OpenSimUDPBase.AsyncEndReceive (System.IAsyncResult iar) [0x0003c] in /home/kunta/opensim-0.9.2/OpenSim/Region/ClientStack/Linden/UDP/OpenSimUDPBase.cs:407
On the CLIENT side (Singularity 8232) the client console shows nothing obviously bad.
NEXT I TP's a second Avatar to the region that first Av is in. To second AV the region appears normal. I flew Second Av to First Av. Second AV sees Fist av ok, who is 'away'. First Av is still stuck in place and does not see Second Av.
Conclusion: The Region simulator is ok, but the connection (UDP at least) between FIrst Av client and simulator has failed.
(0036264)
Kitto Flora (reporter)
2020-03-07 19:38

Further to above:
Searching the Client Console for around the time that Server reported UDP error, I found the following:
2020-03-08T02:32:09Z WARNING: processObjectUpdate: Object update from unknown region! 2804900342602072320
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 1 bytes at position 16659 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 2 bytes at position 16660 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 1 bytes at position 16662 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 2 bytes at position 16663 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 1 bytes at position 16665 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 2 bytes at position 16666 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 1 bytes at position 16668 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 2 bytes at position 16669 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 1 bytes at position 16671 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 2 bytes at position 16672 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 1 bytes at position 16674 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 2 bytes at position 16675 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 1 bytes at position 16677 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 2 bytes at position 16678 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 1 bytes at position 16680 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 2 bytes at position 16681 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 1 bytes at position 16683 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 2 bytes at position 16684 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 1 bytes at position 16686 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: logRanOffEndOfPacket: Ran off end of packet ImprovedTerseObjectUpdate from 74.118.76.133:9116 trying to read 2 bytes at position 16687 going past packet end at 1141
2020-03-08T02:32:09Z WARNING: processObjectUpdate: Object update from unknown region! 2804900342602072320
2020-03-08T02:32:09Z WARNING: processObjectUpdate: Object update from unknown region! 2804900342602072320
(0036265)
tampa (reporter)
2020-03-07 22:58

Those you can log via lludp debug, though you need to uncomment that in code to have access to those commands. UDP problems can come from anywhere, your net, the server, anything in between, hard to diagnose those. The region and server are not overloaded in any way are they?
(0036267)
UbitUmarov (administrator)
2020-03-08 11:43

try current master
(0036268)
Kitto Flora (reporter)
2020-03-08 11:48

On starting client today - connecting to 0.9.2 server, noticed this in client console:
Lots of
INFO: doParse: Unrecognized character while parsing: int(65)

int(65)??? Is this a typo in some xml from server?

Also several of:
WARNING: callbackDecoded: DECODE FAILED: id = b6822e37-faa6-4b19-bc78-eb6bec49e25a, Discard = 0
WARNING: doWork: b6822e37-faa6-4b19-bc78-eb6bec49e25a: Decode of cached file failed (removed), retrying

For tampa:
The region and server are not close to being overloaded. One Av in region. Check on server Memory and CPU use shows plenty of spare.
(0036269)
UbitUmarov (administrator)
2020-03-08 11:53

clear the viwer cache. there maybe something damaged on it
(0036273)
Kitto Flora (reporter)
2020-03-10 15:15

Update and an error:
We now have OpenSim 0.9.2.0 Yeti DreamNation 910aced58a4e421fb981f8afa82afefd71d0cecf 2020-03-09 06:03:52 -0700 (Unix/Mono)
which was built Monday early morning (USA Pacific time) and is locally called '092B'.
I have since had two 3+ hour sessions on the big region with no UDP errors.
092B has also been loaded onto a smaller region, with a 6+ Avatars on it. It has had several crashes and self-restarts. Extract from the simulator console capture:
14:30:16 - [APPLICATION]:
APPLICATION EXCEPTION DETECTED: System.UnhandledExceptionEventArgs

Exception: System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at ....
....
Application is terminating: True

Region (Phoebus) # [ERROR] FATAL UNHANDLED EXCEPTION: System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
at ....

....
14:30:22 - ========================= STARTING OPENSIM =========================

I have the sim console messages saved if you need more detail.

Note for tampa: these regions are running on serious Dell servers: 20CPU cores, typically 80% idle, KiB Mem : 19804574+total, 11115732+free, 21647212 used, 65241208 buff/cache.
(0036275)
tampa (reporter)
2020-03-11 00:03

Can you attach more complete logs of these crashes?
(0036277)
UbitUmarov (administrator)
2020-03-11 00:54

yes those logs provide no information of where the issue happens :(
(0036278)
UbitUmarov (administrator)
2020-03-11 09:49

the crash should be fixed in master now.
(0036283)
Kitto Flora (reporter)
2020-03-16 20:45

On 15th March we ran a region on the latest OS version and held a test party. Some 20 Avs attended. After about 25 minutes the sim crashed. No obvious cause shown in the log. I was watching the sim console during the party - there were a LOT of " # ??[WARNING]: EOC marker not found. Codestream is corrupted." might be associated with texture decode when new Avs enter the region.
We restarted the region and most Avs returned for another 2 hours of partying, no further crashes, and a lot less EOC warnings.
Ubit: I recall you mentioning something about a j2k library(?) mod to an ini file. Is the EOC thing possibly connected to that? Do you want me to try the ini file change, if so be very explicit about what file needs to be changed and how.
(0036284)
UbitUmarov (administrator)
2020-03-17 02:31

try
UseCSJ2K = false
at end of [Startup] on opensim.ini

no idea why viewers keep using lludp to fetch textures...
this is not a sim crash thing, that was elsewhere :(

- Issue History
Date Modified Username Field Change
2020-03-01 14:24 Kitto Flora New Issue
2020-03-01 14:26 Kitto Flora Summary Actually 0.9.2.0 + NPC tweak. Reported on different clients and platforms. => Client view jumps to old location for a few seconds, then back. Repeats.
2020-03-01 14:26 Kitto Flora Additional Information Updated View Revisions
2020-03-01 15:40 UbitUmarov Note Added: 0036236
2020-03-02 22:21 mewtwo0641 Note Added: 0036239
2020-03-02 22:50 UbitUmarov Note Added: 0036240
2020-03-02 22:51 UbitUmarov Note Edited: 0036240 View Revisions
2020-03-02 23:23 tampa Note Added: 0036241
2020-03-03 02:11 mewtwo0641 Note Added: 0036243
2020-03-03 02:20 Luisillo_Contepomi Note Added: 0036244
2020-03-03 02:21 Luisillo_Contepomi Note Edited: 0036244 View Revisions
2020-03-03 03:21 mewtwo0641 Note Added: 0036245
2020-03-04 12:47 Kitto Flora Note Added: 0036247
2020-03-04 13:01 tampa Note Added: 0036248
2020-03-04 13:27 UbitUmarov Note Added: 0036249
2020-03-04 16:42 UbitUmarov Note Added: 0036250
2020-03-04 17:06 Kitto Flora Note Added: 0036251
2020-03-04 17:18 UbitUmarov Note Added: 0036252
2020-03-05 04:26 Luisillo_Contepomi Note Added: 0036253
2020-03-05 04:36 UbitUmarov Note Added: 0036254
2020-03-05 05:35 Luisillo_Contepomi Note Deleted: 0036253
2020-03-05 05:35 Luisillo_Contepomi Note Added: 0036255
2020-03-05 10:46 tampa Note Added: 0036256
2020-03-07 19:23 Kitto Flora Note Added: 0036263
2020-03-07 19:38 Kitto Flora Note Added: 0036264
2020-03-07 22:58 tampa Note Added: 0036265
2020-03-08 11:43 UbitUmarov Note Added: 0036267
2020-03-08 11:48 Kitto Flora Note Added: 0036268
2020-03-08 11:53 UbitUmarov Note Added: 0036269
2020-03-10 15:15 Kitto Flora Note Added: 0036273
2020-03-11 00:03 tampa Note Added: 0036275
2020-03-11 00:54 UbitUmarov Note Added: 0036277
2020-03-11 09:49 UbitUmarov Note Added: 0036278
2020-03-16 20:45 Kitto Flora Note Added: 0036283
2020-03-17 02:31 UbitUmarov Note Added: 0036284


Copyright © 2000 - 2012 MantisBT Group
Powered by Mantis Bugtracker