Chat log from the meeting on 2015-06-23

[11:00] Aine.Caoimhe @refugegrid.com:8002: @Bluewall....can you think of a reason why an OSG visitor specifically entering a region would suddenly cause a ~500MB increase in RAM usage for the simulator that is then subsequently never released? It doesn't happen every time, but it only happens when the visitor is an OSG resident who HGs there
[11:00] Aine.Caoimhe @refugegrid.com:8002: and the only way to free the memory again is to restart the simulator
[11:01] BlueWall Slade: No aine,unless there is something in the networking causing buffers
[11:01] Richardus.Raymaker @hypergrid.org:8002: Linux ?
[11:01] BlueWall Slade: you might look at the stats
[11:01] Nebadon Izumi kommt in Chat-Reichweite (10.69 m).
[11:01] Nebs Metal Bar Stool v1.5 (w/sit & launch): Hello Nebadon Izumi, enjoy your sit..
[11:01] Aine.Caoimhe @refugegrid.com:8002: that was on a linux server, yes
[11:01] BlueWall Slade: see if packet delivery is haning
[11:01] Shez Oyen: Think of the Devil, hi Neb :)
[11:02] BlueWall Slade: Hi Neb
[11:02] Wolf.Freeman @free-world.spdns.de:8002: ~ hallo ~
[11:02] Richardus.Raymaker @hypergrid.org:8002: sofar i know linux is bad in GC.
[11:02] Nebadon Izumi: hello
[11:02] Richardus.Raymaker @hypergrid.org:8002: hi wolf
[11:02] Richardus.Raymaker @hypergrid.org:8002: or better mono is bad
[11:02] Aine.Caoimhe @refugegrid.com:8002: with others the GC seems to work okay for other visitors though
[11:02] BlueWall Slade: Mono runs ok
[11:02] Richardus.Raymaker @hypergrid.org:8002: but 500MB is still to much if it where scripts
[11:02] Aine.Caoimhe @refugegrid.com:8002: scripts are disabled in that region too
[11:03] Alicia.Raven @grid.spellscape.co.uk kommt in Chat-Reichweite (19.45 m).
[11:03] Alicia.Raven @grid.spellscape.co.uk kommt in Chat-Reichweite (18.32 m).
[11:03] Alicia.Raven @grid.spellscape.co.uk: hello :)
[11:03] Aine.Caoimhe @refugegrid.com:8002: and (at least so far) it's only an OSG visitor who tps directly into the region that causes it
[11:03] Shez Oyen: Hi Alicia :)
[11:04] Aine.Caoimhe @refugegrid.com:8002: anyone else is doesn't seem to spike nearly as badly and it clear itself within a couple minutes
[11:04] BlueWall Slade: do they have an unusually complex avatar?
[11:04] Richardus.Raymaker @hypergrid.org:8002: is ther mabye memory cache ?
[11:04] Nebadon Izumi: there is nothing special about how OSG is configured really
[11:04] BlueWall Slade: poor Internet connection?
[11:04] Aine.Caoimhe @refugegrid.com:8002: no....generic SL with a few attachments
[11:04] Nebadon Izumi: Does every OSgrid user do it?
[11:04] Aine.Caoimhe @refugegrid.com:8002: no
[11:04] Nebadon Izumi: or just a paritcular user
[11:04] Aine.Caoimhe @refugegrid.com:8002: and the same user can TP in multiple times and only trigger it on one or two occasions
[11:05] Aine.Caoimhe @refugegrid.com:8002: so it's one of those dreaded hard-to-duplicate bugs
[11:05] Nebadon Izumi: my guess is this person maybe has spotty network connection or something
[11:05] Nebadon Izumi: packet loss lots of reqeuing
[11:05] Aine.Caoimhe @refugegrid.com:8002: they *do* have an enormous inventory on one of the users it has happened with
[11:05] BlueWall Slade: see if UnackB in stats is high
[11:05] Nebadon Izumi: might cause a bottle neck
[11:06] Nebadon Izumi: that would only matter if they teleport prior to it being downloaded
[11:06] Nebadon Izumi: in which case you would probably see errors on your console
[11:06] Aine.Caoimhe @refugegrid.com:8002: mhmm
[11:06] Nebadon Izumi: lots and lots of errors
[11:06] Aine.Caoimhe @refugegrid.com:8002: and there are no errors at all in console
[11:06] Nebadon Izumi: ya my guess is they have packet loss
[11:06] Nebadon Izumi: maybe on wifi connection
[11:06] Aine.Caoimhe @refugegrid.com:8002: it's possible
[11:06] Aine.Caoimhe @refugegrid.com:8002: but it's not jsut one user
[11:07] Nebadon Izumi: is this region hosted at your home?
[11:07] Aine.Caoimhe @refugegrid.com:8002: no
[11:07] Nebadon Izumi: ok
[11:07] Aine.Caoimhe @refugegrid.com:8002: it's on the same box as our main grid server
[11:07] Lucy Afarensis kommt in Chat-Reichweite (19.69 m).
[11:07] Aine.Caoimhe @refugegrid.com:8002: in a datacenter in Montreal
[11:08] Lucy Afarensis kommt in Chat-Reichweite (18.01 m).
[11:08] Nebadon Izumi: yea its really hard to say without just speculating / guessing with so little data to go on
[11:08] Dahlia.Trimble @grid.wwweb3d.net:8002 kommt in Chat-Reichweite (19.76 m).
[11:08] Dahlia.Trimble @grid.wwweb3d.net:8002: hi
[11:08] Aine.Caoimhe @refugegrid.com:8002: yeah
[11:08] Nebadon Izumi: hello Dahlia
[11:08] Alicia.Raven @grid.spellscape.co.uk: hi dahlia
[11:08] Shez Oyen: Hi Dahlia :)
[11:09] Aine.Caoimhe @refugegrid.com:8002: I had read some comments Bluewall made that were posted in a blog from the Safari this week
[11:09] Aine.Caoimhe @refugegrid.com:8002: re profiles
[11:09] Aine.Caoimhe @refugegrid.com:8002: and just wondered if somehow it could be related to that
[11:09] Nebadon Izumi: that seems unlikely
[11:09] Aine.Caoimhe @refugegrid.com:8002: since we're running core profils
[11:09] BlueWall Slade: I really doubt it
[11:09] Nebadon Izumi: profiles are so little amount of data
[11:09] Nebadon Izumi: its probably less than 1mb
[11:10] Aine.Caoimhe @refugegrid.com:8002: it's the size of it too, that's really weird
[11:10] Aine.Caoimhe @refugegrid.com:8002: 500MB is a massive amount to have it suddenly eat
[11:10] BlueWall Slade: look at your network stats
[11:10] Nebadon Izumi: this is under mono right?
[11:10] Aine.Caoimhe @refugegrid.com:8002: yes.....3.4.something I think
[11:10] Nebadon Izumi: we had some issues here on this region and all the plazas using mono 4.0.1 and huge amounts of memory being used
[11:11] Nebadon Izumi: we reverted back to 3.12 on this box and 3.2.3 on lbsa box
[11:11] Robert Adams kommt in Chat-Reichweite (18.48 m).
[11:11] Aine.Caoimhe @refugegrid.com:8002: yes
[11:11] Nebadon Izumi: and problem went away
[11:11] Aine.Caoimhe @refugegrid.com:8002: Tim has the same issue
[11:11] Shez Oyen: Robert :)
[11:11] Nebadon Izumi: and opensim wont even run on master git mono anymore
[11:11] Alicia.Raven @grid.spellscape.co.uk: since i updated on the 18th i noticed cpu useage is higher aswell for some reason
[11:11] Nebadon Izumi: ive been trying once a week for the last month
[11:11] Nebadon Izumi: and it still won't load
[11:12] Robert Adams: hello all... back after being away for a while
[11:12] Dahlia.Trimble @grid.wwweb3d.net:8002: what's the error?
[11:12] BlueWall Slade: Hi Robert
[11:12] Dahlia.Trimble @grid.wwweb3d.net:8002: hi Robert
[11:12] Shez Oyen: WB Robert
[11:12] Nebadon Izumi: let me see if i can find the error
[11:12] Nebadon Izumi: 1 sec
[11:12] Alicia.Raven @grid.spellscape.co.uk: hi robert
[11:13] Andrew Hellershanks kommt in Chat-Reichweite (18.32 m).
[11:13] BlueWall Slade: I've been running ok on whatever the latest packages from Xamarin are, not a heavy load, but really stable.
[11:13] Shez Oyen: Hi Lucy :)
[11:14] Robert Adams: you're discussing problems that seem to be related to using the latest Mono?
[11:14] Nebadon Izumi: http://pastebin.com/6cvydQY1
[11:14] Lucy Afarensis: Hi Shez
[11:14] Nebadon Izumi: that is the error Dahlia
[11:14] Nebadon Izumi: I have been trying their weekly tarball version of mono once a week for the past month Robert
[11:14] Nebadon Izumi: and opensimulator will no longer start
[11:15] Nebadon Izumi: it gets the above message in that pastebin
[11:15] BlueWall Slade: looks like database error?
[11:15] Andrew Hellershanks: Could the 500MB spike be related to someone having a big inventory?
[11:15] Nebadon Izumi: seems like it cant read ini
[11:15] Nebadon Izumi: i dont think so BlueWall
[11:15] BlueWall Slade: I mean that sectino
[11:15] BlueWall Slade: Nini?
[11:16] Nebadon Izumi: I am not sure
[11:16] Nebadon Izumi: if i change back to mono 4.0.1
[11:16] Nebadon Izumi: it starts right up
[11:16] Dahlia.Trimble @grid.wwweb3d.net:8002: Nebadon, looks like maybe a dll loading issue
[11:16] Nebadon Izumi: without making any changes to anything
[11:16] Nebadon Izumi: ya
[11:16] Nebadon Izumi: maybe they changed mono addins or something
[11:16] Nebadon Izumi: technically it shouldnt break anything even if they did
[11:16] BlueWall Slade: will it work with just the default settings using SQLite?
[11:16] Nebadon Izumi: I don't know
[11:17] Nebadon Izumi: I didnt actually try that
[11:17] Nebadon Izumi: but I would be surprised if it does
[11:17] Nebadon Izumi: I dont think it even gets as far as database
[11:17] Andrew Hellershanks: Nebadon, that pastebin looks like what you might get if a (DB related) connection string is incorrect or missing.
[11:17] Nebadon Izumi: it instantly crashes
[11:17] BlueWall Slade: also might look at it under debugger
[11:18] Nebadon Izumi: Andrew usually the error says MySQL
[11:18] Andrew Hellershanks: The code needs a common way to test connection strings after it reads them from the ini so it can report if it is bad.
[11:18] Nebadon Izumi: if it cant read connection string properly
[11:18] Andrew Hellershanks: Nebadon, not always. It doesn't often know about the problem until later.
[11:18] Nebadon Izumi: this error is not specific
[11:18] Nebadon Izumi: it actually is the same exact error you get if you download master git opensim
[11:18] BlueWall Slade: maybe put a little bit of logging in the datbase module to see if it is loading, or if that message is bogus
[11:18] Nebadon Izumi: and rename OpenSim.ini.example
[11:18] Nebadon Izumi: and dont change anything
[11:18] Nebadon Izumi: then start up opensim
[11:18] BlueWall Slade: right
[11:19] Nebadon Izumi: you get the same exact error
[11:19] Nebadon Izumi: which makes it feel like of like maybe its having trouble with Nini
[11:19] Nebadon Izumi: and embeded links to other ini files
[11:19] BlueWall Slade: I think that is one of the first things to get checked.
[11:19] Andrew Hellershanks: You did uncomment on of the architectcure include lines?
[11:20] Nebadon Izumi: yes this region runs constantly
[11:20] Nebadon Izumi: its Sisyphus on OSgrid
[11:20] Nebadon Izumi: all i did was change path to mono
[11:20] Nebadon Izumi: and restart region
[11:20] Nebadon Izumi: it wnt load
[11:20] Nebadon Izumi: change it back to old mono it loads
[11:20] BlueWall Slade: when you try it again, try starting a new standalone with SQLite just to see
[11:20] Nebadon Izumi: ok
[11:20] Nebadon Izumi: I suspect it wont
[11:20] BlueWall Slade: probably not
[11:20] Nebadon Izumi: because it literally crashes after its done listing the ini files
[11:20] Nebadon Izumi: you dont see the simulator start at all
[11:21] Wolf.Freeman @free-world.spdns.de:8002: i have red errors
[11:21] Wolf.Freeman @free-world.spdns.de:8002: too
[11:21] Nebadon Izumi: i'll give that a shot soon though
[11:21] Wolf.Freeman @free-world.spdns.de:8002: LLclientview
[11:22] Wolf.Freeman @free-world.spdns.de:8002: if i opnen my profile
[11:23] BlueWall Slade: Robert, I have been looking into the avatar placement and movement to see about the non-standard avatars.
[11:24] Nebadon Izumi: Wolfman.Freeman ya you are probably running Core Profiles on your grid
[11:24] BlueWall Slade: Looking back, it seems like the start of this was probably changes in the region crossing code back in 2013
[11:24] Nebadon Izumi: OSgrid is still running OSprofiles
[11:25] Nebadon Izumi: so there is some miscommunications between them still
[11:25] Nebadon Izumi: its mostly just noise, nothing to really worry about
[11:25] Wolf.Freeman @free-world.spdns.de:8002: i run core
[11:25] Simulator Version v0.5 ruft: OpenSim 0.8.2.0 Dev        7edaf1d: 2015-06-19 08:44:34 -0700 (Unix/Mono)
[11:25] Wolf.Freeman @free-world.spdns.de:8002: i know
[11:25] Nebadon Izumi: ya
[11:25] Wolf.Freeman @free-world.spdns.de:8002: yeah
[11:26] Wolf.Freeman @free-world.spdns.de:8002: core-profiles works, but red errors makes me nervios
[11:26] Robert Adams: what problems are you looking at, BlueWall?
[11:27] BlueWall Slade: There is a mantis - I'll look for the number in a sc, but small avatars get planted into the ground and causes them to bounce
[11:28] BlueWall Slade: NPC are also different from avatars and need to be adjusted too I guess.
[11:28] Robert Adams: ah... remember that one... these are tinies or just short avatars?
[11:28] BlueWall Slade: Right now, I'm studying to see how they are placed, moved, etc.
[11:29] Dahlia.Trimble @grid.wwweb3d.net:8002: do you know if the capsule bounces also?
[11:29] Robert Adams: I don't know why NPCs are different than avatars.... something must be initialized differently when they two types of avatars are created
[11:29] BlueWall Slade: standard avatars (inventory parts) an a small frame I guess?
[11:29] Robert Adams: I'd suspect the capsule bounces in the viewer and not in-world
[11:29] BlueWall Slade: Dahlia, I don't. How would I tell?
[11:30] BlueWall Slade: IO went back to 0.7.6 and movement is pretty good ...
[11:30] Dahlia.Trimble @grid.wwweb3d.net:8002: not sure, maybe a prim attached to the pelvis point
[11:30] BlueWall Slade: Also stairs are problematic now
[11:30] Nebadon Izumi: the capsule has changed since then
[11:30] Nebadon Izumi: I think
[11:30] Sarah Kline kommt in Chat-Reichweite (14.51 m).
[11:30] Dahlia.Trimble @grid.wwweb3d.net:8002: or perhaps the nametag bouncing
[11:31] BlueWall Slade: I reverted the capsule change and the previous one and that did help.
[11:31] Robert Adams: the viewer tries to keep the avatar above what it is standing on.... if the simulator tells the viewer a height then a location, the viewer tries to use that to keep the avatar above ground in the view
[11:31] Robert Adams: that would make the viewer move the avatar up then the sinulator telling it a different location
[11:31] BlueWall Slade: but, there must be a happy medium, because in 0.7.6 it all worked pretty smooth
[11:32] Dahlia.Trimble @grid.wwweb3d.net:8002: ya the viewer tries to keep the feet on the ground even if the ground is not level
[11:32] Robert Adams: yes,,, changed the capsule (to a box to reduce CPU load) and tweeked the height defaults to make some things better
[11:32] Richardus.Raymaker @hypergrid.org:8002: how are things going to work when the feet offset is in all viewers ?
[11:32] BlueWall Slade: smaller avatars fett are planted into the ground pretty deep.
[11:32] BlueWall Slade: when they move, they bounce, then go back into the ground
[11:33] BlueWall Slade: http://opensimulator.org/mantis/view.php?id=7535 is the mantis.
[11:33] Alicia.Raven @grid.spellscape.co.uk: ive seen that problem but only with tiny mesh avatars
[11:33] Robert Adams: considering SL added height adjustment to the viewer, they know their avatar height/location protocol sucks
[11:34] BlueWall Slade: But, at that point I just used bisect to see the point where all avatars feet were above ground
[11:34] Robert Adams: it is a hard system to make work in all cases
[11:34] BlueWall Slade: I think so :)
[11:34] Richardus.Raymaker @hypergrid.org:8002: and ground sit is still broken
[11:34] Richardus.Raymaker @hypergrid.org:8002: http://opensimulator.org/mantis/view.php?id=6813
[11:34] Nebadon Izumi: ya ground sit seems to ignore prims below you
[11:34] Nebadon Izumi: no matter where you are
[11:34] Nebadon Izumi: you literally sit on the terrain
[11:35] Aine.Caoimhe @refugegrid.com:8002: but not always
[11:35] Aine.Caoimhe @refugegrid.com:8002: :p
[11:36] Aine.Caoimhe @refugegrid.com:8002: but it's really fun when it happens when you're in a skybox a few thousand meters above ground
[11:36] Nebadon Izumi: lag seems to be coming from someone who is messing with their inventory
[11:36] BlueWall Slade: Lately I am setting up different "milestone" versin of OpenSim in VM to test the movement and steps
[11:36] Wolf.Freeman @free-world.spdns.de:8002: ...::: ✱  Goodbye .. وداعا .. Au Revoir .. Arrivederci  ✱
[11:37] BlueWall Slade: bye Wolf
[11:37] Andrew Hellershanks: I'm getting the idea that Wolf is leaving. :)
[11:37] Shez Oyen: lol
[11:37] Richardus.Raymaker @hypergrid.org:8002: bye wolf
[11:38] Alicia.Raven @grid.spellscape.co.uk: any one any ideas what could be causing high cpu load? changes around early this month?
[11:38] Shez Oyen: Sarah.. what is happening to you?
[11:39] Richardus.Raymaker @hypergrid.org:8002: ?
[11:39] Dahlia.Trimble @grid.wwweb3d.net:8002: I haven't noticed any change in load
[11:39] BlueWall Slade: can only use bisect, really, to locate a specific commit.
[11:39] BlueWall Slade: I haven't either.
[11:40] Nebadon Izumi: Can't say I have noticed it either
[11:40] BlueWall Slade: but, I have had scripts misbehave in the past after an upgrade.
[11:40] Alicia.Raven @grid.spellscape.co.uk: might be caused by ubuntu upgrades, there were alof of changes at the time i updated os
[11:40] Dahlia.Trimble @grid.wwweb3d.net:8002: I thought I remember some complaints about the time some of the inventory changes went in but I don't know if they are related
[11:41] Andrew Hellershanks: Alicia, Did you remove the cached copies of compiled scripts?
[11:41] Aine.Caoimhe @refugegrid.com:8002: mhmm
[11:41] Andrew Hellershanks: wow. It takes several seconds for what I type to show up.
[11:41] Andrew Hellershanks: sometimes.
[11:41] Alicia.Raven @grid.spellscape.co.uk: no, i will clear scripts and see if that helps
[11:41] Aine.Caoimhe @refugegrid.com:8002: and there is definitely a much harder hit on an login for a minute or so with them
[11:42] Shez Oyen: lag is very bad today
[11:42] Alicia.Raven @grid.spellscape.co.uk: yea chat lag here also lol
[11:42] Aine.Caoimhe @refugegrid.com:8002: but it doesn't last very long
[11:42] Dahlia.Trimble @grid.wwweb3d.net:8002: ya chat laggggggg
[11:42] Richardus.Raymaker @hypergrid.org:8002: yes it's a bit laggy
[11:42] Shez Oyen: I couldn't even sit.
[11:42] Aine.Caoimhe @refugegrid.com:8002: I bet it's someone with a very latent viewer
[11:42] Aine.Caoimhe @refugegrid.com:8002: this is *exactly* the symptom we see for it
[11:42] Nebadon Izumi: yea not sure what is up
[11:42] Richardus.Raymaker @hypergrid.org:8002: not only chat, seen it with sitting too
[11:42] Nebadon Izumi: i restarted this region about 30 minutes before meeting
[11:42] Dahlia.Trimble @grid.wwweb3d.net:8002: I don't think the new stats are working. I see very low Sim FPS but Time Dilation is 1.0
[11:42] Aine.Caoimhe @refugegrid.com:8002: run a check on UnAckB
[11:42] Nebadon Izumi: i see a ton of timeouts on the console for threading
[11:43] Aine.Caoimhe @refugegrid.com:8002: there will be someone with very, very high count
[11:43] Nebadon Izumi: Timeout detected for thread "AsyncLSLCmdHandlerThread". ThreadState=Background. Last tick was 7473ms ago.
[11:43] Nebadon Izumi: lots of these
[11:43] Lucy Afarensis: I am on a network in my studio building could I be causing the latentcy?
[11:43] Sheera Khan: I have 250 - 750 KByte unack'ed Bytes at times...
[11:43] Dahlia.Trimble @grid.wwweb3d.net:8002: sounds like Wright Plaza is having a high cpu load issue :P
[11:43] Aine.Caoimhe @refugegrid.com:8002: and spawning thousand of resends per minute
[11:43] Nebadon Izumi: cpu is low
[11:43] Richardus.Raymaker @hypergrid.org:8002: week ago one of my regions got complete wonky 100% cpu load. slow request. other simulator displayed mysql error. Still not sure why the region got screwed.
[11:43] Nebadon Izumi: memory is low
[11:43] Alicia.Raven @grid.spellscape.co.uk: i had a day where the sims where showing alot of thread timeouts, it seemed to clear up and i thought it was the datacenter doing work
[11:44] Shez Oyen: it was real slow before anyone else was here.. I arrieved about 10 before the meeting and had trouble flying up here
[11:44] Nebadon Izumi: unacked bytes are low
[11:44] Nebadon Izumi: ya I am pretty sure region is dead though
[11:45] Lucy Afarensis: Had no trouble getting here
[11:45] Nebadon Izumi: physics are sim stats are at 0fps
[11:45] Aine.Caoimhe @refugegrid.com:8002: hmmm....anyone with an insanely high resend count?
[11:45] Nebadon Izumi: and shit ton of thread errors
[11:45] Nebadon Izumi: nope
[11:46] BlueWall Slade: hmm, can you get to the thread names?
[11:46] Nebadon Izumi: how?
[11:46] BlueWall Slade: hehe, I forgot
[11:46] Andrew Hellershanks: ping time is121mS. That seems rather high
[11:46] Richardus.Raymaker @hypergrid.org:8002: oops 5500ms
[11:46] Richardus.Raymaker @hypergrid.org:8002: simulator time is nice brave 5500ms before i hity enter
[11:46] BlueWall Slade: I need to do that again and write a script
[11:46] Richardus.Raymaker @hypergrid.org:8002: seems fine now
[11:46] Dahlia.Trimble @grid.wwweb3d.net:8002: I get 79 here
[11:46] Sheera Khan: 236 ms from Europe
[11:46] BlueWall Slade: but, you have to get the pid of the instance, then dive through proc
[11:47] BlueWall Slade: /proc
[11:47] Richardus.Raymaker @hypergrid.org:8002: ping is 131ms europe
[11:47] Richardus.Raymaker @hypergrid.org:8002: oh what happens if i log file get to big. ?
[11:47] BlueWall Slade: or so I though
[11:47] Nebadon Izumi: seeing tons of inventory posts that are very slow too
[11:47] BlueWall Slade: chat is moving along nicely now
[11:47] Nebadon Izumi: i suspect its inventory
[11:47] Alicia.Raven @grid.spellscape.co.uk: my ping is 164 and im in england
[11:47] Andrew Hellershanks: hm... gone down to about 50mS ping time now.
[11:47] Richardus.Raymaker @hypergrid.org:8002: is your logile 32MB or bigger ?
[11:48] Andrew Hellershanks: nm. Looked at wrong stat. ping time still the same.
[11:48] BlueWall Slade: .
[11:48] Nebadon Izumi: of course it is
[11:48] Nebadon Izumi: the log file would be 32mb in like 10 minutes here
[11:48] Nebadon Izumi: lol
[11:48] BlueWall Slade: haha
[11:48] Nebadon Izumi: I really doubt it has anything to do with log file
[11:48] Dahlia.Trimble @grid.wwweb3d.net:8002: lol cant move
[11:49] Aine.Caoimhe @refugegrid.com:8002: sim and physics fps are all over the place
[11:49] Richardus.Raymaker @hypergrid.org:8002: ok. blame mono ? :P
[11:49] Andrew Hellershanks: They are really low unless this has the corrected numbers.
[11:49] Nebadon Izumi: it seems like the Inventory stuff is coming from Robert
[11:49] Nebadon Izumi: has his uuid in all of the messages
[11:49] Richardus.Raymaker @hypergrid.org:8002: 11fps is max andrew
[11:49] Richardus.Raymaker @hypergrid.org:8002: sofar i know new nuimbers here
[11:50] Andrew Hellershanks: yea, I wasn't sure if that code was in place here.
[11:50] Lucy Afarensis: 48 ping time for me
[11:50] Richardus.Raymaker @hypergrid.org:8002: run robert ! :)
[11:50] Nebadon Izumi: the inventory messages seem to line up with the timeouts
[11:50] Aine.Caoimhe @refugegrid.com:8002: it's dropping down to 0 periodically
[11:50] Nebadon Izumi: im pretty sure whatever is happening is because of inventory requests
[11:50] Richardus.Raymaker @hypergrid.org:8002: dilation 0 second agi, sim / physics same
[11:51] Andrew Hellershanks: sim and phys FPS numbers are ok. mostly sitting at 11
[11:51] Aine.Caoimhe @refugegrid.com:8002: then bouncing up to more than the max 11.3 (how can that happen?)
[11:51] Aine.Caoimhe @refugegrid.com:8002: yes
[11:51] Robert Adams: not sure what that means off hand
[11:51] Robert Adams: the stats I see has physics time low but 'simulation time' gets very large
[11:51] Richardus.Raymaker @hypergrid.org:8002: it's swu=inging between 0 and 11
[11:51] Aine.Caoimhe @refugegrid.com:8002: and those inventory requests somehow manage to flood the queue
[11:51] Nebadon Izumi: i am looking at the console
[11:51] Aine.Caoimhe @refugegrid.com:8002: so no other traffic gets through
[11:51] Robert Adams: I'd think that simulation time is all the calls to avatatr updates, etc
[11:51] Nebadon Izumi: and i see a bunch of very long inventory posts that include your UUID ropert
[11:51] Nebadon Izumi: Robert*
[11:51] Aine.Caoimhe @refugegrid.com:8002: very soon the region will time us all out and kick us
[11:52] Andrew Hellershanks: The meeting is about over anyway so no problem.
[11:52] Nebadon Izumi: that all coincide at the same time the Thread time outs are occuring
[11:52] Aine.Caoimhe @refugegrid.com:8002: unless it manages to recover before that happens
[11:52] Nebadon Izumi: ie:
[11:52] Nebadon Izumi: 18:50:13 - [WATCHDOG]: 9773 ms since Watchdog last ran. Interval should be approximately 2500 ms
18:50:13 - [WATCHDOG]: Timeout detected for thread "AsyncLSLCmdHandlerThread". ThreadState=Background. Last tick was 5614ms ago.
18:50:21 - [WATCHDOG]: 7792 ms since Watchdog last ran. Interval should be approximately 2500 ms
18:50:28 - [WATCHDOG]: 7412 ms since Watchdog last ran. Interval should be approximately 2500 ms
18:50:41 - [LOGHTTP]: Slow SynchronousRestForms request 41466 POST to http://inventory.osgrid.org/xinventory took 26191ms, 0ms writing, METHOD=GETMULTIPLEFOLDERSCONTENT&PRINCIPAL=9ebe6a26-1b31-44ef-89d5-47b79e24fcc4&FOLDERS=00000000-0000-0000-0000-000000000000&COUNT=1
18:50:54 - [LOGHTTP]: Slow SynchronousRestForms request 41526 POST to http://inventory.osgrid.org/xinventory took 10237ms, 0ms writing, METHOD=GETMULTIPLEFOLDERSCONTENT&PRINCIPAL=9ebe6a26-1b31-44ef-89d5-47b79e24fcc4&FOLDERS=00000000-0000-0000-0000-000000000000&COUNT=1
[11:52] Richardus.Raymaker @hypergrid.org:8002: if the region can wait 8 minutes aine
[11:52] Nebadon Izumi: over and over again
[11:52] Dahlia.Trimble @grid.wwweb3d.net:8002: did someone forget to set MONO_THREADS_PER_CPU ?
[11:52] Robert Adams: points to the HTTP server
[11:52] Nebadon Izumi: nope
[11:53] Nebadon Izumi: its only your UUID though Robert
[11:53] Nebadon Izumi: kind of strange
[11:53] Aine.Caoimhe @refugegrid.com:8002: this is the exact problem we reported back in January
[11:53] Nebadon Izumi: there is 1000s of lines of what i posted above
[11:53] Robert Adams: Im using FireStorm, wirelessly
[11:53] Aine.Caoimhe @refugegrid.com:8002: and it is a common occurence when someone enters a region with a highly latent viewer or one that is being throttled
[11:54] Richardus.Raymaker @hypergrid.org:8002: wireless, tssss
[11:54] Dahlia.Trimble @grid.wwweb3d.net:8002: Robert are you at work? I think your work firewall is pretty strict about what it lets through
[11:54] Robert Adams: I haven't looked at my inventory.... wonder what its doing
[11:54] Nebadon Izumi: did you log in here with a cleared cache?
[11:54] Nebadon Izumi: wierd It would completely clobber the sim like this
[11:54] Aine.Caoimhe @refugegrid.com:8002: ,h,,
[11:55] Aine.Caoimhe @refugegrid.com:8002: one viewer can do it;....completely kill a sim
[11:55] Nebadon Izumi: yea seems so heh
[11:55] Robert Adams: I have a ping time of 143ms and am receiving about 40KB/s of UDP traffic
[11:55] Robert Adams: 5KB/s of 'objects'
[11:55] Andrew Hellershanks: I have a test grid running on a machine wtih a firewall. THe grid seems me log in but the viewer never gets the response. Not sure what to do about it as I thought I already had the needed holes punched in the firewall.
[11:56] Andrew Hellershanks: s/seems me login/does log me in/
[11:56] Shez Oyen: maybe a fail safe should be built in to kill any user/client that drains resources
[11:56] Robert Adams: my inventory is still "fetching"
[11:56] Shez Oyen: ahh
[11:56] Richardus.Raymaker @hypergrid.org:8002: that's possible the problem
[11:56] Robert Adams: the number of items is not going up
[11:56] Andrew Hellershanks: Robert, how big is your inventory?
[11:57] BlueWall Slade: looks like we still have a little buglett in there
[11:57] Dahlia.Trimble @grid.wwweb3d.net:8002: Shez, then Nebadon wound never be able to log in with his huge inventory ;)
[11:57] Robert Adams: not humongous... but I've gotten a lot of physis test objects over time
[11:57] BlueWall Slade: .
[11:57] Richardus.Raymaker @hypergrid.org:8002: Dahlia, neb is wiored
[11:57] Richardus.Raymaker @hypergrid.org:8002: wire
[11:57] Shez Oyen: Dahlia :o
[11:58] Lucy Afarensis: My inventory is over 12000 items
[11:58] Andrew Hellershanks: Robert, the type of objects shouldn't matter if you don't rez them.
[11:58] Andrew Hellershanks: that was about 8 to 9 seconds of lag on my previous comment
[11:58] Robert Adams: could be that fetching inventory over a connection with latency causes a timeout failure somewhere which it does't recover from -- in fact it goes into a bad failure modee
[11:58] Lucy Afarensis: me too
[11:58] Richardus.Raymaker @hypergrid.org:8002: Did we not talked about this before when justin where here
[11:59] Robert Adams: I'll log out... be gone for a few minutes... then come back
[11:59] Aine.Caoimhe @refugegrid.com:8002: yes...I've been bringing up this issue since Christmastime
[12:00] Shez Oyen: Haven't seen Justin for a couple of weeks
[12:00] Aine.Caoimhe @refugegrid.com:8002: and it seemed to coincide roughly with the ghost patch merges
[12:00] Lucy Afarensis: We have run into it a lot on the Safari
[12:00] Aine.Caoimhe @refugegrid.com:8002: *ghost brnach
[12:00] BlueWall Slade: A lot of work has gone into the handling of inventory over the past month or so.
[12:00] Aine.Caoimhe @refugegrid.com:8002: it's a lot older issue than that, Bluewall
[12:00] Andrew Hellershanks: Bluewall, true. Could still be one or two minor issues still to iron out
[12:00] Aine.Caoimhe @refugegrid.com:8002: has been happening since December sometime
[12:01] Lucien Lovelace kommt in Chat-Reichweite (14.71 m).
[12:01] Nebadon Izumi: I pasted info into IRC for diva to see as well
[12:01] Lucy Afarensis: I am typing like a mad fiend for no eason
[12:01] Shez Oyen: the lag was bad before Robert arrived
[12:01] Lucy Afarensis: reason
[12:01] BlueWall Slade: haha
[12:01] Nebadon Izumi: its still doing it
[12:01] Nebadon Izumi: even though he left
[12:02] Nebadon Izumi flüstert: its still fetching inventory stuff for his UUID
[12:02] Richardus.Raymaker @hypergrid.org:8002: it need some time to empty the querry i think
[12:02] Andrew Hellershanks: Speaking of leaving, I'm going to do so. Have an electronics thing to work on.
[12:02] Aine.Caoimhe @refugegrid.com:8002: see also: http://opensimulator.org/mantis/view.php?id=7393 and http://opensimulator.org/mantis/view.php?id=7308 and http://opensimulator.org/mantis/view.php?id=7395
[12:02] Nebadon Izumi: here is what i pasted in IRC
[12:02] Richardus.Raymaker @hypergrid.org:8002: bye andrew
[12:02] Nebadon Izumi: sampling
[12:03] Nebadon Izumi: http://pastebin.com/5RCjWza9
[12:03] Andrew Hellershanks: See you next week.
[12:03] Nebadon Izumi: the region does seem a bit smoother now
[12:03] Nebadon Izumi: the console seems quieter all the sudden too now
[12:03] Sheera.Khan @hypergrid.org:8002: one question before everyone leaves: would anyone have the chat-log of last weeks meeting for posting it in the office-hours?
[12:03] Robert Adams kommt in Chat-Reichweite (18.35 m).
[12:03] Nebadon Izumi: doh timeout occured again
[12:03] Jim Jackson: hi RObert
[12:03] Dahlia.Trimble @grid.wwweb3d.net:8002: still laggy
[12:03] Sheera.Khan @hypergrid.org:8002: WB Robert :-)
[12:03] Nebadon Izumi: console seems quieter though
[12:03] Robert Adams: well, did things get better while I was gone?
[12:03] Lucy Afarensis: I think I will split before I get caught in a crash and have spend an hour rebuiding this avatar
[12:04] Aine.Caoimhe @refugegrid.com:8002: it usually takes about 5-10 minutes after the person leaves before it may or may not catch up
[12:04] Robert Adams: was that a thread timeout, Neb?
[12:04] BlueWall Slade: bye Lucy
[12:04] Aine.Caoimhe @refugegrid.com:8002: there will be a massive stack of stuff piled up in the queue
[12:04] BlueWall Slade: .
[12:04] Nebadon Izumi: no robert nothing changed
[12:04] Nebadon Izumi: and there it goes again
[12:04] Nebadon Izumi: the entire time you were gone it was still trying to do inventory queries for you
[12:04] Jayde.Tremor @login.greatcanadiangrid.ca:8002: Hello Jim
[12:04] Nebadon Izumi: and it still is
[12:04] Richardus.Raymaker @hypergrid.org:8002: so, it seems robert is the new test candidate for this bug ?
[12:04] Nebadon Izumi: it did seem to pause a minute but now its doing it again
[12:04] Shez Oyen: I wonder if anyone will address the elephant in the room.. any idea what Albert is announcing in 2 weeks? 2 weeks!?
[12:04] Jim Jackson: hello
[12:04] Robert Adams: oooo... there might be a queue for inventory requests....
[12:04] Lucy Afarensis: bye alldam still here
[12:05] Shez Oyen: Hi Jim :)
[12:05] Jim Jackson: hi Shez
[12:05] Robert Adams: some big annoucement in two weeks?
[12:05] Shez Oyen: he posted about it on the forum board
[12:05] Dahlia.Trimble @grid.wwweb3d.net:8002: those requests really should not be able to slow the simulation
[12:05] Nebadon Izumi: ya nothing to worry about its all good stuff
[12:05] Robert Adams: my inventory stays "Fetching 1,464 Items..." and it is not changing
[12:06] Nebadon Izumi: hmm ok Robert yea something bonkers going on
[12:06] Dahlia.Trimble @grid.wwweb3d.net:8002: did you clear cache?
[12:06] Nebadon Izumi: I think the queue is just flooded
[12:06] Shez Oyen: Thanks for that much Neb :o
[12:06] Aine.Caoimhe @refugegrid.com:8002: @Dahlia: yes....it ignores all throttle settings for the sim too
[12:06] Nebadon Izumi: please just ignore it, its literally 100% false
[12:06] Nebadon Izumi: I know there has been some kind of Notecard with rumors going around
[12:07] Robert Adams: could be a run away retry loop
[12:07] Nebadon Izumi: thats kind of what it looks like Robert
[12:07] Nebadon Izumi: I doubt its going to recover heh
[12:07] Nebadon Izumi: the console is a mess
[12:08] Nebadon Izumi: I need to run out for about an hour, I will try to contact diva in a bit see maybe what she things might be going on
[12:08] Shez Oyen: Robert.. the lag started before you got here, I don't think it's all yer fault.