[11:00] Nebadon Izumi: hello Justin
[11:00] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Hi JC
[11:00] Connecting to in-world Voice Chat...
[11:00] Connected
[11:00] Justin Clark-Casey: hi bluewall, folks
[11:00] Marcus.Llewellyn @grid.verymad.net:8002: Mainly, it looks like possibly a nicer alternative to the current remote admin stuff.
[11:00] Nebadon Izumi: Mic messaged me earlier on IRC and said he would be coming in today
[11:00] BlueWall.Slade @gateway.bluewallvirtual.net:8002: cool
[11:00] Nebadon Izumi: its much more than that Marcus
[11:01] Nebadon Izumi: Mic wrote dispatcher to interface with SUMO
[11:01] Nebadon Izumi: a Traffic Simulator
[11:01] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Yes, it's a scene manipulator
[11:01] Justin Clark-Casey: ok. Did you ree-enable adaptive throttles?
[11:01] Marcus.Llewellyn @grid.verymad.net:8002: I know... it can do LOADS. But that's mmy primary interest for the momet.
[11:01] Nebadon Izumi: oh probably not Justin
[11:01] Nebadon Izumi: he literally just messaged me
[11:01] Justin Clark-Casey: ok, I'm assuming that's what he would like to see tested. See if we can reproduce the problems we were seeing
[11:01] Marcus.Llewellyn @grid.verymad.net:8002: Kinda thought it could do very neat things when talking to node.js.
[11:01] Nebadon Izumi: and to be honest there are far more people here today than I expected there to be
[11:02] Aine.Caoimhe @refugegrid.com:8002: we were hoping for Christmas rum cake
[11:02] Richardus.Raymaker @hypergrid.org:8002: its really time you add the : in the slurl
[11:02] Richardus.Raymaker @hypergrid.org:8002: cc.opensimulator.org:8002 Staff Zone 1
[11:02] Nebadon Izumi: I suppose i could flip over Keynotes quickly
[11:02] Marcus.Llewellyn @grid.verymad.net:8002: The cake is a lie.
[11:02] Justin Clark-Casey: yeah, didn't expect you to be here
[11:02] Nebadon Izumi: we could always jump over
[11:02] Justin Clark-Casey: yeah, flipping the keynotes would be good
[11:02] Nebadon Izumi: ok what do I change again?
[11:02] Justin Clark-Casey: In ClientStack.LindenUDP, set enable_adaptive_throttles = true
[11:03] Justin Clark-Casey: or just uncomment whatever it's set to, since true is the default
[11:03] Dahlia.Trimble @grid.wwweb3d.net:8002 is Online
[11:03] Nebadon Izumi: ok cool wasnt sure if there was more to it
[11:03] Mic Bowman is Online
[11:04] Kayaker.Magic @grid.kitely.com:8002: I have been researching how to make weapons that use llCastray,
[11:04] Kayaker.Magic @grid.kitely.com:8002: and I've found a bunch of problems with it
[11:04] Nebadon Izumi: ok its restarting
[11:04] Richardus.Raymaker @hypergrid.org:8002: hope you did not shoot in your own feet
[11:04] Nebadon Izumi: hello Mic
[11:05] Mic Bowman: hi nebadon
[11:05] Kayaker.Magic @grid.kitely.com:8002: who works n llCastray?
[11:05] Justin Clark-Casey: kayaker: yeah, I saw your bug reports
[11:05] Justin Clark-Casey: hello mic
[11:05] Richardus.Raymaker @hypergrid.org:8002: i looked once at it.. but still puzzled
[11:05] Nebadon Izumi: hmm I don't think anyone has worked on llCastRay in a while
[11:05] Mic Bowman: hi justin
[11:05] Nebadon Izumi: was that Teravus?
[11:05] Richardus.Raymaker @hypergrid.org:8002: and i looked in SL at it
[11:05] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Grrr.... I have to go see to the cat. He is being insistent he wants to see me.
[11:05] Justin Clark-Casey: yes, but then Melanie I believe
[11:05] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: bbiab
[11:05] Mic Bowman: i've updated a bunch of the dispatcher docs in github for you
[11:06] Richardus.Raymaker @hypergrid.org:8002: Andrew, let the cat come to you
[11:06] Justin Clark-Casey: from looking at the bugs, they all looked potentially fixable but the time I have spent on opensim recently has been absorbed in some script attachment issues
[11:06] Justin Clark-Casey: mic: thanks, haven't read your e-mails yet - haven't been online since last Fri
[11:07] Justin Clark-Casey: I'm just being a hardass about things as usual
[11:07] Nebadon Izumi: lol
[11:07] Mic Bowman: no rush justin... adding a few things
[11:07] Dahlia.Trimble @grid.wwweb3d.net:8002 is Online
[11:07] Mic Bowman: and good forcing function for documentation
[11:07] Dahlia.Trimble @grid.wwweb3d.net:8002: hi
[11:07] Nebadon Izumi: hello Dahlia
[11:07] Justin Clark-Casey: my presence is going to be quite spotty certainly until after xmas
[11:07] Justin Clark-Casey: hi dahlia
[11:08] Marcus.Llewellyn @grid.verymad.net:8002: We demand that you work through the holidays.
[11:08] Richardus.Raymaker @hypergrid.org:8002: this month is just a mess
[11:08] Justin Clark-Casey: I do have to get into gear again in the new year, not been feeling it recently
[11:08] Nebadon Izumi: :)
[11:09] BlueWall.Slade @gateway.bluewallvirtual.net:8002: brown-out?
[11:09] Justin Clark-Casey: yeah, a bit
[11:09] Seth.Nygard @refugegrid.com:8002: You are likely still a bit burned out from all the work for OSCC
[11:09] Billy.Bradshaw @hg.viewtwo.net:8600: I have been testing mono 3.12 against 2 linux kernels 3.17 and 3.22 and as yet seen no issues running Opensim.
[11:09] Justin Clark-Casey: I need to look for client work again, and maybe make more of a patronage push, I'm thinking of doing patreon
[11:10] Nebadon Izumi: neat
[11:10] Nebadon Izumi: never heard of that before
[11:11] Justin Clark-Casey: Becoming more popular, I have to change my policy of asking people to just post to ml when they contact me though :)
[11:11] Justin Clark-Casey: but I much, much prefer public talk on tech issues
[11:11] Marcus.Llewellyn @grid.verymad.net:8002: It looks geared mainly toward artists?
[11:11] Justin Clark-Casey: yeah, but a few software people use it. Can't so a 'cash per commit' model though :)
[11:11] Kayaker.Magic @grid.kitely.com:8002: And what of the art of proramming?
[11:12] Aine.Caoimhe @refugegrid.com:8002: we had another rash of very odd issues at a party last night....a region that was running smooth as silk with about a dozen people on it, then when 1 particular person tped into it, the sim ground to a halt with an absolute flood of packet resends....in one case it actually crashed the region....the second time it caused the region to temporarily lock up and then boot everyone who was there but it didn't actually crash.....scary that a single person can bring a sim down like that
[11:12] Nebadon Izumi: I think Plugh was mentioning a similar problem Aine
[11:12] Aine.Caoimhe @refugegrid.com:8002: yes
[11:13] Nebadon Izumi: I havent seen anything like that myself
[11:13] Aine.Caoimhe @refugegrid.com:8002: we've seen it increasingly often in the last little while
[11:13] BlueWall.Slade @gateway.bluewallvirtual.net:8002: they have a bad internet connection?
[11:13] Dahlia.Trimble @grid.wwweb3d.net:8002: http resends?
[11:13] Aine.Caoimhe @refugegrid.com:8002: and not just in our own
[11:13] Aine.Caoimhe @refugegrid.com:8002: saw the same thing at Metro last Friday at Close Encounter
[11:13] Justin Clark-Casey: were you planning on doing any work on this, Mic?
[11:13] Mic Bowman: that sounds like the throttle problem again
[11:13] Aine.Caoimhe @refugegrid.com:8002: and then again in AVW
[11:13] Mic Bowman: yes
[11:13] Mic Bowman: but...
[11:13] Dahlia.Trimble @grid.wwweb3d.net:8002: I saw a similar problem with wrong plaza but it was http
[11:13] Nebadon Izumi: generally the throttle problems we experienced were isolated to a single person or just a few people
[11:14] Mic Bowman: given that the throttle code has been in there for a LONG time
[11:14] Aine.Caoimhe @refugegrid.com:8002: Seth has the logs for it all
[11:14] Dahlia.Trimble @grid.wwweb3d.net:8002: I dont think packet resends get echoed on the console
[11:14] Nebadon Izumi: infact Mic, i had 2 viewers running from same computer connected to same simulator
[11:14] Nebadon Izumi: and one got throttled the other did not
[11:14] Mic Bowman: i'm wondering what happened recently that is causing these problems
[11:14] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: back
[11:14] Mic Bowman: i have a feeling we are seeing a "cliff" problems
[11:14] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Aine - are the visitors using grid accounts - or hypergrid?
[11:14] Mic Bowman: something changed recently... that much seems certain
[11:14] Seth.Nygard @refugegrid.com:8002: processed packet #3273, type ParcelPropertiesRequest from 111shawn.resident @craft-world.org
[11:14] Seth.Nygard @refugegrid.com:8002: The messages all are in this form.2014-12-22 18:58:08,207 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1531, type AgentAnimation from 111shawn.resident @craft-world.org
2014-12-22 18:59:20,162 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #3270, type ParcelPropertiesRequest from 111shawn.resident @craft-world.org
2014-12-22 18:59:20,163 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #3271, type ParcelPropertiesRequest from 111shawn.resident @craft-world.org
2014-12-22 18:59:20,163 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #3272, type ParcelPropertiesRequest from 111shawn.resident @craft-world.org
2014-12-22 18:59:20,163 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Receive
[11:15] Justin Clark-Casey: mic: I know what you're saying, but I'm often surprised to hear new about problems that are actually long standing. It could be increasing use and ppl talking about this is making it more visible
[11:15] Mic Bowman: and now with latency issues...
[11:15] Dahlia.Trimble @grid.wwweb3d.net:8002: but throttles are on a per client basia arent they?
[11:15] Mic Bowman: stuff is piling up
[11:15] Justin Clark-Casey: mic: or, I (or someone else) could have unwittingly inserted a bug
[11:15] Aine.Caoimhe @refugegrid.com:8002: it was a mix.....vast majority of the people in the region were local
[11:15] Aine.Caoimhe @refugegrid.com:8002: then there were maybe 4-5 HG visitors
[11:15] Justin Clark-Casey: I would be surprised if it were throttles as they are per client
[11:15] Aine.Caoimhe @refugegrid.com:8002: the one who caused the issues was a HG user
[11:15] BlueWall.Slade @gateway.bluewallvirtual.net:8002: did you pin it to a particular visitor?
[11:15] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Oh, ok
[11:15] Aine.Caoimhe @refugegrid.com:8002: but I've see then exact same thing with a local user too
[11:15] Mic Bowman: justin... i've been spending quite a bit of time looking at the code
[11:15] Nebadon Izumi: Diva said she found some interesting things with Inventory requests, though i dont think that is the case here
[11:15] Mic Bowman: there is a linear search through the resend queue
[11:16] Justin Clark-Casey: seth: I am still waiting for someone to record a stat trace when these issues happen
[11:16] Alicia.Raven @grid.spellscape.co.uk: ive seen this problem going back along time, just dont get people with bad connections come in that oftern
[11:16] Mic Bowman: if that queue gets big
[11:16] Mic Bowman: and its being searched with a lock
[11:16] Justin Clark-Casey: mic: yes, though thinking about it further, I think even a big queue wouldn't delay things much... but that is speculatino
[11:16] Mic Bowman: then everyone would see crappy performance
[11:16] Seth.Nygard @refugegrid.com:8002: I've been trying to get some data during one of these events, but so far nothing useful
[11:16] Dahlia.Trimble @grid.wwweb3d.net:8002: yes, its as if a hg person coming in with many assets from a place with a slow or buggy asset server will kill the sim with http requests
[11:16] Mic Bowman: i think clearly we can making the estimator longer
[11:16] Aine.Caoimhe @refugegrid.com:8002: exactly, Dahlia
[11:16] Justin Clark-Casey: the hg asset requests are not throttled in master
[11:16] Mic Bowman: given the error messages we're seeing
[11:17] Justin Clark-Casey: are now
[11:17] Nebadon Izumi: brb someone at my door
[11:17] Aine.Caoimhe @refugegrid.com:8002: and maybe we're just seeing it so often now because there's so many more people HGing around these days
[11:17] Dahlia.Trimble @grid.wwweb3d.net:8002: so its not udp packets, its http
[11:17] Mic Bowman: what? hg asset requests are throttled?
[11:17] BlueWall.Slade @gateway.bluewallvirtual.net:8002: In cases like that - would it be better to use microthreadding?
[11:17] Justin Clark-Casey: mic: yes
[11:17] Mic Bowman: that's not the adaptive throttle code
[11:17] Mic Bowman: that adaptive throttles are solely for client connections
[11:17] Justin Clark-Casey: yes, that's entirely different...
[11:18] Mic Bowman: hmm...
[11:18] Justin Clark-Casey: personally, I think it's impossible to say without rerproducing the probelm or getting a stat trace which shows an identifiable issue
[11:18] Mic Bowman: right... but there seems to be in the mantis entries some combination of low latency connections
[11:18] Mic Bowman: and hg connections
[11:19] Aine.Caoimhe @refugegrid.com:8002: I have a feeling it's probably a case where no one thing triggers it...that perhaps it's a case where two borderline things happen to both occur at the same time
[11:19] Dahlia.Trimble @grid.wwweb3d.net:8002 longs for the good old days when network code that used multiple connects used select()...
[11:19] Mic Bowman: all three entires i've been following mention it
[11:19] Billy.Bradshaw @hg.viewtwo.net:8600: The grid owner from the party last night said adaptive throttles had been disabled.
[11:19] Seth.Nygard @refugegrid.com:8002: I tried to replicate the problem with only adding high latency but could not cause the same issue in controlled conditions
[11:19] Mic Bowman: seth: if you figure it out please let me know
[11:19] Mic Bowman: i can't reproduce it either
[11:19] Justin Clark-Casey: seth: how did you add high latency?
[11:20] Seth.Nygard @refugegrid.com:8002: In our case last night, I know adaptive throttles were enabled
[11:20] Aine.Caoimhe @refugegrid.com:8002: when it does, general chat suddenly becomes horribly laggy, scripts stop responding, you see serious time dilation (0.6 or worse) and sim fps and physics fps drop from the usual 55 down into the 20's and 30's
[11:20] Mic Bowman: been trying to figure out how to get a network emulator to add latency
[11:20] Nebadon Izumi: back
[11:20] Alicia.Raven @grid.spellscape.co.uk: i was wondering if firestorms debug settings can simulate lost packets
[11:20] Justin Clark-Casey: if actually scene fps changes then it's unlikely to be anything to do with throttles
[11:20] Mic Bowman: seth: please do a show throttles and show queues
[11:20] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Linux TC
[11:20] Mic Bowman: next time that happens
[11:20] Mic Bowman: and post to mantis
[11:20] Mic Bowman: or send to me
[11:20] Seth.Nygard @refugegrid.com:8002: I am routing through a linux box, as using the traffic control to add latency, with the netem module
[11:20] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Aine, the issue nebaon said I had reported to him recently was related to Warp3D. Not sure if if is the same/similar issue to what you have experienced
[11:20] Mic Bowman: i'll bump up the priority of adding the logging
[11:21] Justin Clark-Casey: seth: on all udp packets?
[11:21] Dahlia.Trimble @grid.wwweb3d.net:8002: its like the same effect of teleporting to a sim with an empty sim-side asset cache
[11:21] Seth.Nygard @refugegrid.com:8002: I initially set for all packets, as a wost case
[11:21] Justin Clark-Casey: seth: how much of a delay?
[11:21] Aine.Caoimhe @refugegrid.com:8002: it just seems like thread lock of a flooded queue, but none of the debug reports shows anything
[11:21] Mic Bowman: dahlia... that suggests that there is some resource sharing between client connections and asset requests
[11:21] Justin Clark-Casey: aine: as I said, someone needs to record a stat trace
[11:21] Seth.Nygard @refugegrid.com:8002: 500ms +/- 450 with a normal distribution
[11:22] Mic Bowman: there was a bug in older version of mono where the number of http requests outstanding could shut down a sim
[11:22] Dahlia.Trimble @grid.wwweb3d.net:8002: Mic, yes, I suspect its threadpool related
[11:22] Seth.Nygard @refugegrid.com:8002: That was to try and replicate what I see from out user with the satellite link
[11:22] Mic Bowman: threadpool starvation could cause packet handling to be slow... which could push us over the cliff on the resends
[11:23] Mic Bowman: one thing to look at... again from the mantis entires...
[11:23] Dahlia.Trimble @grid.wwweb3d.net:8002: or cause everything to be slow
[11:23] Justin Clark-Casey: if someone records stats then tp starvation will be seen
[11:23] Mic Bowman: its the resend queue size that gets big
[11:23] Mic Bowman: thats not object updates
[11:23] Mic Bowman: which are (or at least were) put back into the task queue
[11:23] Aine.Caoimhe @refugegrid.com:8002: the one person last night managed to do 8k resends in less than the first 5 minutes of being in the sim....everyone else had been there for 60+ minutes and was under 2k total resends
[11:23] Dahlia.Trimble @grid.wwweb3d.net:8002: console is usually spewing http messages
[11:23] Mic Bowman: so that we resend current state of the object rather than old state
[11:24] Mic Bowman: one quick fix would be to combine the resend and task queues
[11:24] Nebadon Izumi: I was able to trigger similar behavior by connecting like 60+ radegast clients to the load tests
[11:24] Justin Clark-Casey: I hope this isnt' going to be some stabbing in the dark....
[11:24] Justin Clark-Casey: :)
[11:24] Mic Bowman: cause the viewer sets the resend queue to some incredibly low number
[11:25] Nebadon Izumi: my router seems to start floundering around 60 clients
[11:25] Mic Bowman: justin: it is... but at least this one has some data behind it, look at the places in mantis where we have "show queues"
[11:25] Mic Bowman: the resend queue is huge
[11:25] Justin Clark-Casey: which mantises?
[11:25] Aine.Caoimhe @refugegrid.com:8002: the other interesting thing that Seth discovered the other day is the sim appears to completely ignore the viewer's throttle setting....a viewer set to 250kbs will still get sent at far higher rate...whatever the region server thinks it can spit out
[11:26] Mic Bowman: let me get it
[11:26] Aine.Caoimhe @refugegrid.com:8002: not sure if that could be in any way related
[11:26] Nebadon Izumi: even with the throttle it can burst much higher
[11:26] Nebadon Izumi: takes a bit for the throttle to kick in
[11:26] Justin Clark-Casey: seth: what code are you using?
[11:26] Mic Bowman: the throttles do accommodate bursts by design
[11:26] Richardus.Raymaker @hypergrid.org:8002: sorry, distracted. what issue with warp3d ?
[11:27] Aine.Caoimhe @refugegrid.com:8002: stock built from git within the last 2 weeks on both region and grid
[11:27] Justin Clark-Casey: aine: are you and seth operating the same simulators?
[11:27] Aine.Caoimhe @refugegrid.com:8002: Seth is running the grid server and some of the region servers, I'm hosting some additional regions of my own
[11:27] Seth.Nygard @refugegrid.com:8002: The region last night was running code fro, Dec 10
[11:28] Aine.Caoimhe @refugegrid.com:8002: gut the main ones where we hold parties (or expect high traffic) are all in the same network as the grid server
[11:28] Mic Bowman: justin: #7308
[11:28] Seth.Nygard @refugegrid.com:8002: Last night the region server is in a different data center than the grid server
[11:28] Mic Bowman: though it looks like byes unacked
[11:28] Mic Bowman: so it might still be task queue
[11:28] Justin Clark-Casey: mic: that screenshot shows on 2 packets queued for resend, though many resent
[11:28] Mic Bowman: also... it looks like acks are coming back
[11:29] Mic Bowman: given the number of packets in
[11:29] Mic Bowman: which suggests that we aren't waiting long enough for the acks
[11:29] Mic Bowman: either thread problem (not handling them fast enough)
[11:29] Mic Bowman: or the RTT is broken
[11:29] Justin Clark-Casey: that's possible. though it seemd to me any more than a few seconds isn't worth acking anyway?
[11:29] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Seth, I saw issues in a grid when the servers were in two different locations.
[11:30] Mic Bowman: agreed but 13K unacked means that the window is pretty large
[11:30] Mic Bowman: or was
[11:30] Justin Clark-Casey: that is bytes unacked
[11:30] Justin Clark-Casey: not packets
[11:30] Mic Bowman: correct
[11:30] Justin Clark-Casey: 13k doesn't seeem that much
[11:30] Mic Bowman: thats still 15+ packets
[11:30] Mic Bowman: with multiple updates per packet
[11:31] Justin Clark-Casey: really, this shows up much more in the resent number
[11:31] Justin Clark-Casey: pCampbot running many bots triggers the same thing, because pCampbot doesn't process received packets quickly enough, I think
[11:31] Mic Bowman: its both
[11:32] Mic Bowman: unacked are packets that have been put on the wire
[11:32] Justin Clark-Casey: we didn't suffer such issue with pCampbot stuff, which is massively latent
[11:32] Justin Clark-Casey: though we were never running for more than 2 hours
[11:32] Mic Bowman: not disagreeing
[11:32] Mic Bowman: just hard to imagine that after five years of running this code, this is the first time we've had problems
[11:32] Justin Clark-Casey: tbh, it doesn't surprise me
[11:32] Mic Bowman: the throttle code is about that old
[11:33] Mic Bowman: we didn't see it at last years OSCC did we?
[11:33] Justin Clark-Casey: many times I have seen problems which were new to me but had existed for years, just nobody talked about them
[11:33] Mic Bowman: 2013
[11:33] Seth.Nygard @refugegrid.com:8002: I do have a wireshark capture of one night we was this issue, but not the corresponding stats log.
[11:33] Mic Bowman: seth: can you post that to one of the mantis entires (assuming that there is nothing confidential in it)
[11:33] Justin Clark-Casey: mic: last year's oscc it's honestly hard to say. The numbers were pretty low and everyone was sitting
[11:34] Justin Clark-Casey: seth: why didn't you capture thre stats log?
[11:34] Mic Bowman: i'll just say that given the severity... i'm surprised it has been brought up
[11:34] Seth.Nygard @refugegrid.com:8002: It is a pretty large capture file. I can send it anyone that wants to look at it
[11:34] Mic Bowman: and the consistency with which its happening
[11:34] Seth.Nygard @refugegrid.com:8002: That was before we mentioned the need for the stats log.
[11:35] Justin Clark-Casey: seth: ok, because with the stats log I can produce a set of graphs
[11:35] Seth.Nygard @refugegrid.com:8002 nods
[11:35] Justin Clark-Casey: and it will be very obvious if there is tp exhaustion, for instance
[11:35] Seth.Nygard @refugegrid.com:8002: I have some stats logs, but nothing so far that coincides with a large number of issues
[11:35] Aine.Caoimhe @refugegrid.com:8002: the number of HG travelers it up pretty dramatically recently, if that's somehow a contributing factor...it's certainly something I've been experiencing inworld far more in the last 3-4 months than ever before
[11:36] Nebadon Izumi: could be Aine
[11:36] Justin Clark-Casey: aine: yes, I wouldn't be surprised if that isn't playing a role
[11:36] Justin Clark-Casey: I believe it did for oscc, for instance
[11:36] Aine.Caoimhe @refugegrid.com:8002 nods
[11:36] Mic Bowman: HG shouldn't affect the adaptive throttles unless its triggering something
[11:36] Justin Clark-Casey: yeah, I actually doubt this is a throttles problem
[11:37] Mic Bowman: something is triggering a fatal behavior in the throttles
[11:37] Aine.Caoimhe @refugegrid.com:8002: it seems like it needs a perfect storm of conditions to trigger it, but those are being achieved a lot more often recently
[11:37] Seth.Nygard @refugegrid.com:8002: I have tried regions with and without adaptive throttles and see the same behaviour when this happens
[11:37] Mic Bowman: strongly agree aine!
[11:37] Aine.Caoimhe @refugegrid.com:8002: which is what makes me thing HG is at least an element of it
[11:37] Justin Clark-Casey: mic: If it were throttles, I would not expect the scene fps to fall
[11:37] Mic Bowman: correct
[11:37] Aine.Caoimhe @refugegrid.com:8002: mesh might be another since a lot more people are wearing mesh than used to be the case
[11:37] Justin Clark-Casey: which is what some are reporting
[11:38] Justin Clark-Casey: and another thing that would show up in stats traces....
[11:38] Nebadon Izumi: we saw something ineresting on MOSES grid
[11:38] Mic Bowman: unless we are spending so much tim processing resends that we aren't getting other useful work done
[11:38] Nebadon Izumi: which doesnt quite apply but maybe
[11:38] Justin Clark-Casey: mic: that only happens on one thread
[11:38] Nebadon Izumi: since MOSES isnt HG enabled, we are having to move over content via IAR and OAR
[11:38] BlueWall.Slade @gateway.bluewallvirtual.net:8002: ++JC, can't argue with good instrumentation
[11:38] Nebadon Izumi: but these IAR and OAR objects tend to retain the last owner UUID
[11:38] Nebadon Izumi: which somtimes puts their grid into a looping lookup request
[11:38] Nebadon Izumi: that it cant ever satisfy
[11:39] Justin Clark-Casey: neb: how do you know?
[11:39] Aine.Caoimhe @refugegrid.com:8002: yes, I've see than one too (and logged a mantis on it)
[11:39] Marcus.Llewellyn @grid.verymad.net:8002: I've seen what Neb is describing on my little grid, after importing my OSgrid IAR.
[11:39] Nebadon Izumi: been looking for perforamce issues with Doug
[11:39] Nebadon Izumi: we did a small load test yesterday
[11:39] Nebadon Izumi: and that was one of the things taht popped out at him
[11:39] Justin Clark-Casey: neb: how does he know?
[11:39] Mic Bowman: so... i'll add more logging code justin... i want to see what percentage of packets are never acked vs percentage that are acked late
[11:39] Nebadon Izumi: i wasnt watching consoles, mostly relaying what he said
[11:39] Justin Clark-Casey: neb: really difficult to trust second hand info
[11:40] Justin Clark-Casey: mic: ok, it would be great if you could add that to existing stats data....
[11:40] Nebadon Izumi: i suspect in that case anyway justin its more of a simiangrid issue
[11:40] Aine.Caoimhe @refugegrid.com:8002: @Neb: http://opensimulator.org/mantis/view.php?id=7382
[11:40] Mic Bowman: might need some pointers for how to do that
[11:40] Mic Bowman: but sure
[11:40] Mic Bowman: sounds like the right thing to do
[11:41] Marcus.Llewellyn @grid.verymad.net:8002: I get a huge a mount of spew like the following:
[11:41] Marcus.Llewellyn @grid.verymad.net:8002: 2014-10-16 12:52:49,615 DEBUG - OpenSim.Region.CoreModules.Framework.UserManagement.UserManagementModule [USER MANAGEMENT MODULE]: No grid user found for 192523fa-57e3-274e-48c7-a6a0adaeaa81
[11:41] Justin Clark-Casey: sure. At the top of LLUDPServer it shows how a bunch are registered. But very happy to talk about it more when necessary
[11:41] Justin Clark-Casey: mic: if it's in that framework it can then also be recorded, etc.
[11:41] Seth.Nygard @refugegrid.com:8002: Other than good stats and logs, what other data can I collect that is helpful to track these issues down?
[11:42] Mic Bowman: thanks justin... i'll start there & send email if i have questions
[11:42] Aine.Caoimhe @refugegrid.com:8002: console login to Justin so he can watch it live :p
[11:42] Justin Clark-Casey: seth: For me, it's all about the stats record at this stage. That gives actual insight into the state of the system.
[11:42] Justin Clark-Casey: I used this a lot when looking into oscc problems
[11:42] Seth.Nygard @refugegrid.com:8002 nods
[11:42] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Marcus, I have seen a couple of instances where a record in the user accounts table would go missing.
[11:43] Marcus.Llewellyn @grid.verymad.net:8002: These are users that don't exist on my grid. They're likely OSgrid UUIDs.
[11:43] Marcus.Llewellyn @grid.verymad.net:8002: I have a log section where that goes on for around 120 lines.
[11:43] Justin Clark-Casey: this user management stuff is getting complicated
[11:44] BlueWall.Slade @gateway.bluewallvirtual.net:8002: It should give a good hones try to find them, then give up.
[11:44] BlueWall.Slade @gateway.bluewallvirtual.net:8002: honest*
[11:44] Justin Clark-Casey: bluewall: i thought it did cache negatives
[11:44] Marcus.Llewellyn @grid.verymad.net:8002: It takes about two minutes, and bogs the sim while it's happening.
[11:44] Justin Clark-Casey: but doesn't sound like it
[11:45] Mic Bowman: whats the timeout on the cache
[11:45] BlueWall.Slade @gateway.bluewallvirtual.net:8002: It seems that some conditions can make it go for a long time.
[11:45] Mic Bowman: and do we still do those lookups for permission checks?
[11:45] Aine.Caoimhe @refugegrid.com:8002: maybe that's just long enough for it to also stack up a queue of other stuff that has issue if it doesn't get processed rapidly enough
[11:45] Mic Bowman: at one point every login tried to retrieve user ids for all object owners for permission checks
[11:46] Justin Clark-Casey: mic: looking right now at the UMM module, the m_UserCache is just a dictionary with no timeout
[11:46] Marcus.Llewellyn @grid.verymad.net:8002: My asset cache has a 48 hour timeout.
[11:46] Dahlia.Trimble @grid.wwweb3d.net:8002: maybe they just time out because the asset server cant fulfill the requests at the rate the sim asks for them?
[11:46] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I put a manual command to invalidate it last week
[11:46] Justin Clark-Casey: that thing is a bit of a mess
[11:46] Mic Bowman: that's asset cache... what about user cache?
[11:46] Aine.Caoimhe @refugegrid.com:8002: @Dahlia: or is throttling them
[11:46] BlueWall.Slade @gateway.bluewallvirtual.net:8002: In some things, needt top kick a sim over
[11:46] Marcus.Llewellyn @grid.verymad.net:8002: I'm uncertain where to find that.
[11:47] Justin Clark-Casey: mic: I'm not sure what you're asking
[11:47] Mic Bowman: nevermind
[11:47] Mic Bowman: i just know that we got killed on group lookups for permissions
[11:48] Justin Clark-Casey: There's a lot of awful ad-hoc stuff
[11:48] Mic Bowman: had to add a simian specific cache for non-existant users and groups
[11:48] Marcus.Llewellyn @grid.verymad.net:8002: The messages I'm talking about seem to coincide with inventory loading on the client.
[11:49] Aine.Caoimhe @refugegrid.com:8002: yeah...inventory load is another huge issue....impossible to log in to any region not hosted on the same network as the grid server without inventory fetch hanging infinitely and also brining the entire region to its knees
[11:49] Justin Clark-Casey: would also be good to see a stats trace on current behaviuor of that too
[11:49] BlueWall.Slade @gateway.bluewallvirtual.net:8002: si that on other networks besides OSGrid?
[11:50] Justin Clark-Casey: from point of login through to where such issues manifest
[11:50] Aine.Caoimhe @refugegrid.com:8002: yes....happens on ours
[11:50] Aine.Caoimhe @refugegrid.com:8002: I can't log in to any region I host
[11:50] Marcus.Llewellyn @grid.verymad.net:8002: On my little one man grid as well.
[11:50] Seth.Nygard @refugegrid.com:8002: I've seen it on several grids
[11:50] Justin Clark-Casey: marcus: are you able to generate a stats trace for that?
[11:50] BlueWall.Slade @gateway.bluewallvirtual.net:8002: I don't see it here, but I'm all in one network.
[11:50] Seth.Nygard @refugegrid.com:8002: I can generate stats for that as well
[11:51] Justin Clark-Casey: ok, thanks
[11:51] Aine.Caoimhe @refugegrid.com:8002: (I can, but it will then sit there generating 1000+ HTTP fetches per minute until I log our again and then takes up to an hour to finally process the queue even after I've logged out
[11:51] Marcus.Llewellyn @grid.verymad.net:8002: I have to admit I've never done that. I'm happy to do it, though.
[11:51] Aine.Caoimhe @refugegrid.com:8002: Bluewall and I spent a Sunday doing tests on that in the spring
[11:51] Marcus.Llewellyn @grid.verymad.net:8002: I'm pretty sure logging in with a clean viewer cache repeats the issue for me.
[11:51] Justin Clark-Casey: marcus: http://opensimulator.org/wiki/Show_stats#stats_record
[11:51] Justin Clark-Casey: is is the case that whatever else, we should remove pending requests if the user leaves the region
[11:52] Aine.Caoimhe @refugegrid.com:8002: I don't even need to clear cache...happen on every single login
[11:52] Marcus.Llewellyn @grid.verymad.net:8002: Ah, didn't know about that functionality. Thanks. Will do that. :)
[11:52] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Maaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[11:52] Nebadon Izumi: lol
[11:52] BlueWall.Slade @gateway.bluewallvirtual.net:8002: sorry about that
[11:53] Dahlia.Trimble @grid.wwweb3d.net:8002 blames the cat for sitting on the keyboard
[11:53] Marcus.Llewellyn @grid.verymad.net:8002: Is there an extant mantis for this issue?
[11:53] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Marcus - you grid. Is it on several servers?
[11:53] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Hey, it wasn't my cat. He doesn't do that
[11:53] Justin Clark-Casey: marcus; the inventory one?
[11:54] Marcus.Llewellyn @grid.verymad.net:8002: My grid has two machine right now. Robust/Mysql is on one. The simulators are on another.
[11:54] Marcus.Llewellyn @grid.verymad.net:8002 nods to JCC
[11:54] BlueWall.Slade @gateway.bluewallvirtual.net:8002: ok, in-house?
[11:54] Aine.Caoimhe @refugegrid.com:8002: http://opensimulator.org/mantis/view.php?id=7054
[11:54] Justin Clark-Casey: marcus: http://opensimulator.org/mantis/view.php?id=7054
[11:54] Marcus.Llewellyn @grid.verymad.net:8002: Yes, all on the same LAN.
[11:54] BlueWall.Slade @gateway.bluewallvirtual.net:8002: ok, thanks
[11:54] Marcus.Llewellyn @grid.verymad.net:8002: Thanks, jcc. :)
[11:54] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: I've heard of another odd cache(?) related issue as of a few days ago but it seems to involve Warp3D
[11:54] Justin Clark-Casey: warp3d is a memory leaking mess
[11:55] Justin Clark-Casey: internally
[11:55] Nebadon Izumi: this is different Justin
[11:55] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: yeah, I've heard that. The person using it has it set to run every 6 hours.
[11:55] Dahlia.Trimble @grid.wwweb3d.net:8002: does warp3d leak on windows also?
[11:55] Nebadon Izumi: this seems to be causing sim lag for him when someone pulls up the map
[11:55] Nebadon Izumi: which is odd, and it doesnt happen when warp3d is off
[11:55] Justin Clark-Casey: dahlia: don't know
[11:55] Nebadon Izumi: I don't get that at all
[11:55] BlueWall.Slade @gateway.bluewallvirtual.net:8002: maybe the textures are more detailed/larger?
[11:56] Marcus.Llewellyn @grid.verymad.net:8002 was pondering using the dispatcher with an externalised version of Warp3D. :)
[11:56] Nebadon Izumi: that seems unlikely to me
[11:56] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Run grid using Warp3D. Login to grid. First time user calls up map, the instance/region locks upfor about 60 seconds. Console doesn't show any errors other than missed heartbeat. Next person in region to pull up map for first time after login will cause same issue.
[11:56] Nebadon Izumi: we would be seeing texture fetching causing more problems if that were the case I would think
[11:56] Dahlia.Trimble @grid.wwweb3d.net:8002: Ive seen a few cases where code that creates many small objects leaks on mono but not windows
[11:56] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Those tiles are served from Robust
[11:56] Justin Clark-Casey: dahlia: this is a leak due to failure to dispose of graphics objects, I believe
[11:56] Dahlia.Trimble @grid.wwweb3d.net:8002: as if they dont free on mono
[11:56] Nebadon Izumi: right bringing up the map shouldnt invoke warp3d in any way should it?
[11:57] Nebadon Izumi: very strange
[11:57] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Warp3d will send the tiles to Robust when they are created. After that, it's nothing to do with the regin
[11:57] Marcus.Llewellyn @grid.verymad.net:8002: GDI+ on Mono has always been... hinky.
[11:57] Justin Clark-Casey: dahlia: ordinary tile generator was leaking the same way until I fixed it to properly clean up afterwards
[11:57] Justin Clark-Casey: if you have IDiposable objects you need to dispose of them
[11:57] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Robust, and I would have expected them map tiles to cached in Robust. When using standard map tile generator no lockup of region/instance is being seen so it appears related specifically to Warp3D
[11:57] Justin Clark-Casey: it's not GDI's fault, it's bad programming
[11:58] Richardus.Raymaker @hypergrid.org:8002: Not that i know. map use maptiles and the are sitting on the rovbust disk.
[11:58] Nebadon Izumi: lately I have been generating my own tiles with the viewer a screenshot and photoshop
[11:58] Nebadon Izumi: and hard coding tiles
[11:58] Nebadon Izumi: looks so much nicer and less stress on the simulator
[11:58] Justin Clark-Casey: labour intensive :)
[11:58] Richardus.Raymaker @hypergrid.org:8002: so.. how can warp3d screw things when you open map ?
[11:59] BlueWall.Slade @gateway.bluewallvirtual.net:8002: It wouldn't hurt to run Warp3d on startup.
[11:59] Nebadon Izumi: I don't know Richardus, that doest make much sense
[11:59] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Nebadon, I'd like to know how you take the pictures. Might be possible to automate the post capture proccess
[11:59] Justin Clark-Casey: you guys can speculate all you want, it's meaningless until someone looks at the code
[11:59] Dahlia.Trimble @grid.wwweb3d.net:8002: justin, ya I guess failure to dispose (not using "using") could do it also
[11:59] Richardus.Raymaker @hypergrid.org:8002: Bluewall, i only run it at startup
[11:59] Nebadon Izumi: i sit on a cube
[11:59] Nebadon Izumi: and move it to certain coordinates
[11:59] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Warp3D does run on reegion startup to generate maps.
[11:59] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Nebadon - Mesh terrains make bad map tiles, lol
[11:59] Nebadon Izumi: then go into mouseloook
[11:59] Richardus.Raymaker @hypergrid.org:8002: Hands justin a plate with spaghetti :)
[11:59] Marcus.Llewellyn @grid.verymad.net:8002: I use so much mesh now... my maptiles are just big blocks
[11:59] Nebadon Izumi: yes but with mesh terrain
[11:59] Justin Clark-Casey: richardus: I'm just being grouchy :)
[11:59] Nebadon Izumi: its super easy to take your texture
[11:59] Nebadon Izumi: on the mesh and apply it as a map tile hard coded
[12:00] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Nebadon, you could probably script a camera to do that
[12:00] Nebadon Izumi: yea probably
[12:00] Marcus.Llewellyn @grid.verymad.net:8002: +1 Bluewall
[12:00] Aine.Caoimhe @refugegrid.com:8002: render it in Blender
[12:00] Nebadon Izumi: this is how LL does it
[12:00] Nebadon Izumi: they made a custom viewer
[12:00] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Nebadon, send me the coords sometime and I'll try it out.
[12:00] Justin Clark-Casey: richardus: but yeah, the reason I haven't simply fixed warp3d to dispose properly is because it's spaghetti like
[12:00] Nebadon Izumi: that takes snapshots and crawls the grid
[12:00] Justin Clark-Casey: the objects are passed all around the place so it's not easy to track and dispose of them consistently
[12:00] Richardus.Raymaker @hypergrid.org:8002: justin :)
[12:00] Justin Clark-Casey: though even best effort would probably be better than the situation now
[12:00] BlueWall.Slade @gateway.bluewallvirtual.net:8002: LL updates the map tiles every few months
[12:01] Nebadon Izumi: I think it was like 128/128/300
[12:01] Nebadon Izumi: or 128/128/320
[12:01] Nebadon Izumi: something like that id have to look on other grid
[12:01] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: justin, i've tried looking at warp3d code but I don't feel I know enough about object garbage collection in C# to find the source of the leak, nor have I found a way to test to see if the code is leaking.
[12:01] Nebadon Izumi: most of the tiles on this grid are all hard coded screenshots i made
[12:02] Justin Clark-Casey: andrew: I't simply because the IDisposable graphics objects are never Disposed, I expect
[12:02] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Nebadon, I'll send you a tool to help script a camera
[12:02] Nebadon Izumi: search for Keynote on map
[12:02] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Nebadon, ok. Did you have to rescale after in PS?
[12:02] Nebadon Izumi: yes
[12:02] Justin Clark-Casey: that's certainly the case, but I couldn't definitively say that's the whole story until at least that is fixed
[12:02] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Nebadon, ok
[12:02] Nebadon Izumi: it needs to be the # of pixels to # of meters
[12:02] Dahlia.Trimble @grid.wwweb3d.net:8002: should just rip the warp and oar code out and make a program that builds map tiles out of an oar file
[12:02] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Justin, I may have another look. I can't test code that easily right now with my main desktop being dead.
[12:02] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Mic, thanks for the Dispatcher docs. They look good.
[12:02] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Might as well spend some time digging through code.
[12:03] Marcus.Llewellyn @grid.verymad.net:8002: If an automatced crawler is desired, Radegast's 3D view is prolly good enough for it, and that client is easy to modify.
[12:03] Justin Clark-Casey: mic: did you want to try and informal load test now?
[12:03] Justin Clark-Casey: seeing as we are at the hour
[12:03] Mic Bowman: bluewall: i've moved everything to a new git repository
[12:03] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: One other quick question before we all go "poof" for the week
[12:03] Mic Bowman: github.com/cmickeyb/dispatcher
[12:03] Nebadon Izumi: Marcus, but then you don't get shadows and windlight
[12:03] Nebadon Izumi: it would look like poo in Radegast
[12:03] BlueWall.Slade @gateway.bluewallvirtual.net:8002: yep, that's what I'm loking at.
[12:04] Mic Bowman: ok
[12:04] Marcus.Llewellyn @grid.verymad.net:8002: True, but from a couble hundred meters up, that doesn't realy make a dramatic difference anyway. And it still would beat the map generators we got now.
[12:04] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Is there extra call/processing to handle things like ${Const|PrivatePort} in ini settings? Seems one INI doesn't handle some settings.
[12:04] Dahlia.Trimble @grid.wwweb3d.net:8002: do you get shadows and windlight in warp?
[12:04] Mic Bowman: justin: anything we can get is helpful
[12:04] Mic Bowman: though i would prefer to get the stats added
[12:04] BlueWall.Slade @gateway.bluewallvirtual.net:8002: No Andrew
[12:04] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Either XBakes or Profiles doesn't like that in the URL
[12:04] Mic Bowman: can we shoot for next tuesday?
[12:04] Justin Clark-Casey: what do you think Neb
[12:04] Justin Clark-Casey: ?
[12:04] Nebadon Izumi: sure
[12:05] Nebadon Izumi: I am available
[12:05] Mic Bowman: trying to finish dispatcher work before moving to throttles
[12:05] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Andrew, I use it throughout.
[12:05] Mic Bowman: (given your comment about not being able to work on it for awhile)
[12:05] Nebadon Izumi: lol
[12:05] Nebadon Izumi: who's snowballing me
[12:05] Aine.Caoimhe @refugegrid.com:8002: have a great week and a merry Christmas, everyone :)
[12:05] Justin Clark-Casey: merry xmas Aine
[12:06] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Thanks Aine, same you you.
[12:06] Marcus.Llewellyn @grid.verymad.net:8002: B'bye Aine :)
[12:06] Mic Bowman: merry christas!
[12:06] Seth.Nygard @refugegrid.com:8002: yes, Merry Christmas to everyone
[12:06] Dahlia.Trimble @grid.wwweb3d.net:8002: bye Aine :)
[12:06] Justin Clark-Casey: mic, neb: ok, next week then.
[12:06] Dahlia.Trimble @grid.wwweb3d.net:8002: Merry Christmas all :D
[12:06] Nebadon Izumi: cool, sounds like a plan
[12:06] Aine.Caoimhe @refugegrid.com:8002 waves and *poofs*
[12:06] Richardus.Raymaker @hypergrid.org:8002: merry christmas
[12:06] Mic Bowman: sonds good justin
[12:06] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Merry Christmas!
[12:06] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Marcus, I wouldn't try digging through use of Radegast to look at having it crawl to do map tiles as viewer code is C++ and I don't do C++
[12:06] Nebadon Izumi: ya Merry Christmas everyone, thanks for coming, great turn out today
[12:06] Nebadon Izumi: awesome meeting
[12:06] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Merry Christmas everyone.
[12:06] Marcus.Llewellyn @grid.verymad.net:8002: Radegst is C#.
[12:06] Justin Clark-Casey: I'm pretty sure radegast is c#
[12:06] Aine.Caoimhe @refugegrid.com:8002: lol...nope, can't tp back home
[12:07] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Oh, ok. I tend to assume most viewer related stuff is C++ as they usually are.
[12:07] Aine.Caoimhe @refugegrid.com:8002: why does it see me on my internal IP http://opensimulator.org/mantis/view.php?id=7054
[12:07] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: Most are based on common code base coming from SL
[12:07] BlueWall.Slade @gateway.bluewallvirtual.net:8002: haha what is that?
[12:07] Marcus.Llewellyn @grid.verymad.net:8002: I tend to regard Radegast at the real documentation for how to use libOMV in a client. Hehe.
[12:07] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: :)
[12:07] Richardus.Raymaker @hypergrid.org:8002: homework for nebadon. better chairs. or a parachute
[12:07] Aine.Caoimhe @refugegrid.com:8002: 15:06:01 - [SCENE PRESENCE]: Completing movement of Aine Caoimhe into region Aine Workshop in position <128.025, 127.891, 25.419>
15:06:01 - [SCENE]: User Client Verification for Aine Caoimhe in Aine Workshop returned false
15:06:01 - [CLIENT]: Close has been called for Aine Caoimhe attached to scene Aine Workshop
15:06:01 - [SCENE]: Removing child agent Aine Caoimhe 12a7488f-d08c-448e-93ff-6ae67500f709 from Aine Workshop
15:06:01 - [CAPS]: Remove caps for agent 12a7488f-d08c-448e-93ff-6ae67500f709 in region Aine Workshop
15:06:01 - [CAPS]: Unauthorized CAPS client 12a7488f-d08c-448e-93ff-6ae67500f709 from 192.168.1.1:64227
[12:07] Nebadon Izumi: Redagest Rendering is cool
[12:07] Nebadon Izumi: but it has issues
[12:07] Billy.Bradshaw @hg.viewtwo.net:8600: anie, may need to clear hyperlinks
[12:08] Richardus.Raymaker @hypergrid.org:8002: some chairs are a trap !
[12:08] Justin Clark-Casey: aine: that sometimes means the user agent service on this installation could not be contacted
[12:08] Seth.Nygard @refugegrid.com:8002: take care everyone, have a great day
[12:08] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: I'll have the user with the warp3d issue check the Roust console when the reigon locks up
[12:08] Richardus.Raymaker @hypergrid.org:8002: ok. lets see if we can TP home. bye all
[12:08] Aine.Caoimhe @refugegrid.com:8002: kk...guess I log out and back in again then
[12:08] Justin Clark-Casey: aine: one moment
[12:08] Seth.Nygard @refugegrid.com:8002 waves and departs
[12:08] Aine.Caoimhe @refugegrid.com:8002: kk
[12:08] Seth.Nygard @refugegrid.com:8002 is Online
[12:08] Andrew.Hellershanks @login.greatcanadiangrid.ca:8002: oh, yeah. I wonder if I can get back to where I came from
[12:09] Marcus.Llewellyn @grid.verymad.net:8002: I ain't claiming Radegast will give you the fidelity the LL viewer can. It sure won't. But it fits into the "good enough" category.
[12:09] Richardus.Raymaker @hypergrid.org:8002: dont lean back nebadon. theres a snowball
[12:09] Dahlia.Trimble @grid.wwweb3d.net:8002: bye all :) *ctrl-shift-h*
[12:09] Nebadon Izumi: im actually in my home region :)
[12:09] Marcus.Llewellyn @grid.verymad.net:8002: B'bye Dahlia
[12:09] Aine.Caoimhe @refugegrid.com:8002 shouts: my initial TP contact was this: 15:05:59 - [SCENE]: Region Aine Workshop told of incoming child agent Aine Caoimhe 12a7488f-d08c-448e-93ff-6ae67500f709 (circuit code 1108777924, IP 173.33.240.158, viewe
r Firestorm-Releasex64 4.6.9.42969, teleportflags (ViaHome, ViaHGLogin), position <127.9989, 127.9999, 25.96604>. From region Staff Zone 1 (75a5b0cf-9d1d-4f30-8dcf-b25de6
eeae64) @ http://cc.opensimulator.org:8002/
[12:09] Mic Bowman: bye all
[12:09] Dahlia.Trimble @grid.wwweb3d.net:8002 is Online
[12:09] Nebadon Izumi: see you Mic
[12:09] Marcus.Llewellyn @grid.verymad.net:8002: Cya Mic
[12:09] Aine.Caoimhe @refugegrid.com:8002: so in that one I have the correct IP
[12:09] Mic Bowman is Offline
[12:09] Aine.Caoimhe @refugegrid.com:8002: then when it does the verification is has somehow changed to my log IP
[12:10] Justin Clark-Casey: aine: tihs is going back to your home grid?
[12:10] Aine.Caoimhe @refugegrid.com:8002: this is going back to a home-hosted region on our grid, yes
[12:10] Aine.Caoimhe @refugegrid.com:8002: ie the region is one I host, not that the grid hosts
[12:10] Justin Clark-Casey: aine: the failure point is when your simulator attempts to contact your own user agent service for verification and fails for some reason
[12:10] Justin Clark-Casey: I would initially suspect some bug in opensim tbh
[12:11] Justin Clark-Casey: http://opensimulator.org/wiki/Hypergrid_Protocol#Protocol_Flow
[12:11] Justin Clark-Casey: a failure on step 6.5
[12:11] BlueWall.Slade @gateway.bluewallvirtual.net:8002: Take care everyone. See you all next wek.
[12:11] BlueWall.Slade @gateway.bluewallvirtual.net:8002 is Online
[12:11] Aine.Caoimhe @refugegrid.com:8002: yeps
[12:11] Aine.Caoimhe @refugegrid.com:8002: that looks like what's happening
[12:12] Aine.Caoimhe @refugegrid.com:8002: so when it tries to do that, it's seeing my LAN IP not my external facing IP
[12:12] Aine.Caoimhe @refugegrid.com:8002: or actually it's seeing my router's LAN IP
[12:12] Justin Clark-Casey: aine: it might be something like that. I need to put in yet more debugging info, though the current structure of code makes that hard
[12:13] Aine.Caoimhe @refugegrid.com:8002: I think I can tp back to a grid-hosted region though....just not one of my own
[12:13] Aine.Caoimhe @refugegrid.com:8002: much like I can't log in to any region I host
[12:13] Dahlia.Trimble @grid.wwweb3d.net:8002 is Offline
[12:13] Aine.Caoimhe @refugegrid.com:8002: (without fetch bug killing it)
[12:14] Aine.Caoimhe @refugegrid.com:8002: anyway...such is life...I'll just always go via a grid-hosted region
[12:14] Justin Clark-Casey: aine: could you try the tp back again?
[12:14] Aine.Caoimhe @refugegrid.com:8002: to my own one?
[12:14] Alicia.Raven @grid.spellscape.co.uk: gotta go, bye everyone
[12:14] Justin Clark-Casey: yes please
[12:14] Justin Clark-Casey: bye alicia
[12:14] Alicia.Raven @grid.spellscape.co.uk: merry christmas all
[12:14] Aine.Caoimhe @refugegrid.com:8002: kk...here goes
[12:15] Justin Clark-Casey: 192.95.30.185
[12:15] Justin Clark-Casey: is that familiar at all?
[12:15] Justin Clark-Casey: 46.23.253.143
[12:15] Justin Clark-Casey: 173.33.240.158
[12:15] Aine.Caoimhe @refugegrid.com:8002: um....trying to think if it might be our grid's
[12:16] Aine.Caoimhe @refugegrid.com:8002: my outward facing one is the 173.xxx
[12:16] Marcus.Llewellyn @grid.verymad.net:8002: I"m gonna scoot.
[12:16] Marcus.Llewellyn @grid.verymad.net:8002 smiles and waves. :)
[12:16] Aine.Caoimhe @refugegrid.com:8002: bye Marcus
[12:17] Justin Clark-Casey: yeah, something is going on but I think the ultimate failure may be when your own simulator contacts your region's user agent service to verify that the transport is ok
[12:18] Justin Clark-Casey: I can see that IP when this region contacts your region to tell it that the agent is coming
[12:18] Aine.Caoimhe @refugegrid.com:8002: somewhere in there it's switching over to using my router's LAN IP instead of my outward facing one, then of course that is a mismatch to the original IP the tp request came from
[12:19] Justin Clark-Casey: actually, from the trace you IM'd it looks like that might not be getting through
[12:19] Seth.Nygard @refugegrid.com:8002 is Offline
[12:19] Justin Clark-Casey: aine: I dunno. I would say add a mantis but you know how many are currently on the pile
[12:19] Aine.Caoimhe @refugegrid.com:8002: I know
[12:19] Justin Clark-Casey: I am interested in making it easier to diagnose such problems in the long run
[12:20] Seth.Nygard @refugegrid.com:8002 is Online
[12:20] Justin Clark-Casey: or fix them if they are opensim bugs
[12:20] Aine.Caoimhe @refugegrid.com:8002: and it's not the end of the world which is why I haven't made a mantis on it... i know you already have too much on your plate
[12:20] Justin Clark-Casey: ok, let's keep it in mind though. Do feel free to raise sometihng if you want. Might be good just to keep track
[12:20] Aine.Caoimhe @refugegrid.com:8002: the big ones for me at the animation bug, the inventory fetch bug, and the killing region bug since they affect everyone
[12:21] Justin Clark-Casey: yes. I still need to look at the anim bug, my time got caught up with some other issue. The inventory fetch also needs attention
[12:21] Justin Clark-Casey: ok
[12:21] Aine.Caoimhe @refugegrid.com:8002: that one-user killing it for everyone is the worst killer
[12:21] Justin Clark-Casey: yes, hoepfully we can get some progress on that soon
[12:22] Aine.Caoimhe @refugegrid.com:8002: okies....have a great Christmas, Justin....see you next week :)
[12:22] Justin Clark-Casey: ok, I'm going to skeddale. Bye for now
[12:22] Justin Clark-Casey: hope you have a great xmas too Aine
