Chat log from the meeting on 2014-12-23

From OpenSimulator

Jump to: navigation, search
[11:00]  Nebadon Izumi: hello Justin
[11:00]  BlueWall.Slade Hi JC
[11:00]  Connecting to in-world Voice Chat...
[11:00]  Connected
[11:00]  Justin Clark-Casey: hi bluewall, folks
[11:00]  Marcus.Llewellyn Mainly, it looks like possibly a nicer alternative to the current remote admin stuff.
[11:00]  Nebadon Izumi: Mic messaged me earlier on IRC and said he would be coming in today
[11:00]  BlueWall.Slade cool
[11:00]  Nebadon Izumi: its much more than that Marcus
[11:01]  Nebadon Izumi: Mic wrote dispatcher to interface with SUMO
[11:01]  Nebadon Izumi: a Traffic Simulator
[11:01]  BlueWall.Slade Yes, it's a scene manipulator
[11:01]  Justin Clark-Casey: ok. Did you ree-enable adaptive throttles?
[11:01]  Marcus.Llewellyn I know... it can do LOADS. But that's mmy primary interest for the momet.
[11:01]  Nebadon Izumi: oh probably not Justin
[11:01]  Nebadon Izumi: he literally just messaged me
[11:01]  Justin Clark-Casey: ok, I'm assuming that's what he would like to see tested. See if we can reproduce the problems we were seeing
[11:01]  Marcus.Llewellyn Kinda thought it could do very neat things when talking to node.js.
[11:01]  Nebadon Izumi: and to be honest there are far more people here today than I expected there to be
[11:02]  Aine.Caoimhe we were hoping for Christmas rum cake
[11:02]  Richardus.Raymaker its really time you add the : in the slurl
[11:02]  Richardus.Raymaker Staff Zone 1
[11:02]  Nebadon Izumi: I suppose i could flip over Keynotes quickly
[11:02]  Marcus.Llewellyn The cake is a lie.
[11:02]  Justin Clark-Casey: yeah, didn't expect you to be here
[11:02]  Nebadon Izumi: we could always jump over
[11:02]  Justin Clark-Casey: yeah, flipping the keynotes would be good
[11:02]  Nebadon Izumi: ok what do I change again?
[11:02]  Justin Clark-Casey: In ClientStack.LindenUDP, set enable_adaptive_throttles = true
[11:03]  Justin Clark-Casey: or just uncomment whatever it's set to, since true is the default
[11:03]  Dahlia.Trimble is Online
[11:03]  Nebadon Izumi: ok cool wasnt sure if there was more to it
[11:03]  Mic Bowman is Online
[11:04]  Kayaker.Magic I have been researching how to make weapons that use llCastray,
[11:04]  Kayaker.Magic and I've found a bunch of problems with it
[11:04]  Nebadon Izumi: ok its restarting
[11:04]  Richardus.Raymaker hope you did not shoot in your own feet
[11:04]  Nebadon Izumi: hello Mic
[11:05]  Mic Bowman: hi nebadon
[11:05]  Kayaker.Magic who works n llCastray?
[11:05]  Justin Clark-Casey: kayaker: yeah, I saw your bug reports
[11:05]  Justin Clark-Casey: hello mic
[11:05]  Richardus.Raymaker i looked once at it.. but still puzzled
[11:05]  Nebadon Izumi: hmm I don't think anyone has worked on llCastRay in a while
[11:05]  Mic Bowman: hi justin
[11:05]  Nebadon Izumi: was that Teravus?
[11:05]  Richardus.Raymaker and i looked in SL at it
[11:05]  Andrew.Hellershanks Grrr.... I have to go see to the cat. He is being insistent he wants to see me.
[11:05]  Justin Clark-Casey: yes, but then Melanie I believe
[11:05]  Andrew.Hellershanks bbiab
[11:05]  Mic Bowman: i've updated a bunch of the dispatcher docs in github for you
[11:06]  Richardus.Raymaker Andrew, let the cat come to you
[11:06]  Justin Clark-Casey: from looking at the bugs, they all looked potentially fixable but the time I have spent on opensim recently has been absorbed in some script attachment issues
[11:06]  Justin Clark-Casey: mic: thanks, haven't read your e-mails yet - haven't been online since last Fri
[11:07]  Justin Clark-Casey: I'm just being a hardass about things as usual
[11:07]  Nebadon Izumi: lol
[11:07]  Mic Bowman: no rush justin... adding a few things
[11:07]  Dahlia.Trimble is Online
[11:07]  Mic Bowman: and good forcing function for documentation
[11:07]  Dahlia.Trimble hi
[11:07]  Nebadon Izumi: hello Dahlia
[11:07]  Justin Clark-Casey: my presence is going to be quite spotty certainly until after xmas
[11:07]  Justin Clark-Casey: hi dahlia
[11:08]  Marcus.Llewellyn We demand that you work through the holidays.
[11:08]  Richardus.Raymaker this month is just a mess
[11:08]  Justin Clark-Casey: I do have to get into gear again in the new year, not been feeling it recently
[11:08]  Nebadon Izumi: :)
[11:09]  BlueWall.Slade brown-out?
[11:09]  Justin Clark-Casey: yeah, a bit
[11:09]  Seth.Nygard You are likely still a bit burned out from all the work for OSCC
[11:09]  Billy.Bradshaw I have been testing mono 3.12 against 2 linux kernels 3.17 and 3.22 and as yet seen no issues running Opensim.
[11:09]  Justin Clark-Casey: I need to look for client work again, and maybe make more of a patronage push, I'm thinking of doing patreon
[11:10]  Nebadon Izumi: neat
[11:10]  Nebadon Izumi: never heard of that before
[11:11]  Justin Clark-Casey: Becoming more popular, I have to change my policy of asking people to just post to ml when they contact me though :)
[11:11]  Justin Clark-Casey: but I much, much prefer public talk on tech issues
[11:11]  Marcus.Llewellyn It looks geared mainly toward artists?
[11:11]  Justin Clark-Casey: yeah, but a few software people use it. Can't so a 'cash per commit' model though :)
[11:11]  Kayaker.Magic And what of the art of proramming?
[11:12]  Aine.Caoimhe we had another rash of very odd issues at a party last night....a region that was running smooth as silk with about a dozen people on it, then when 1 particular person tped into it, the sim ground to a halt with an absolute flood of packet one case it actually crashed the region....the second time it caused the region to temporarily lock up and then boot everyone who was there but it didn't actually crash.....scary that a single person can bring a sim down like that
[11:12]  Nebadon Izumi: I think Plugh was mentioning a similar problem Aine
[11:12]  Aine.Caoimhe yes
[11:13]  Nebadon Izumi: I havent seen anything like that myself
[11:13]  Aine.Caoimhe we've seen it increasingly often in the last little while
[11:13]  BlueWall.Slade they have a bad internet connection?
[11:13]  Dahlia.Trimble http resends?
[11:13]  Aine.Caoimhe and not just in our own
[11:13]  Aine.Caoimhe saw the same thing at Metro last Friday at Close Encounter
[11:13]  Justin Clark-Casey: were you planning on doing any work on this, Mic?
[11:13]  Mic Bowman: that sounds like the throttle problem again
[11:13]  Aine.Caoimhe and then again in AVW
[11:13]  Mic Bowman: yes
[11:13]  Mic Bowman: but...
[11:13]  Dahlia.Trimble I saw a similar problem with wrong plaza but it was http
[11:13]  Nebadon Izumi: generally the throttle problems we experienced were isolated to a single person or just a few people
[11:14]  Mic Bowman: given that the throttle code has been in there for a LONG time
[11:14]  Aine.Caoimhe Seth has the logs for it all
[11:14]  Dahlia.Trimble I dont think packet resends get echoed on the console
[11:14]  Nebadon Izumi: infact Mic, i had 2 viewers running from same computer connected to same simulator
[11:14]  Nebadon Izumi: and one got throttled the other did not
[11:14]  Mic Bowman: i'm wondering what happened recently that is causing these problems
[11:14]  Andrew.Hellershanks back
[11:14]  Mic Bowman: i have a feeling we are seeing a "cliff" problems
[11:14]  BlueWall.Slade Aine - are the visitors using grid accounts - or hypergrid?
[11:14]  Mic Bowman: something changed recently... that much seems certain
[11:14]  Seth.Nygard  processed packet #3273, type ParcelPropertiesRequest from 111shawn.resident
[11:14]  Seth.Nygard The messages all are in this form.2014-12-22 18:58:08,207 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #1531, type AgentAnimation from 111shawn.resident
2014-12-22 18:59:20,162 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #3270, type ParcelPropertiesRequest from 111shawn.resident
2014-12-22 18:59:20,163 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #3271, type ParcelPropertiesRequest from 111shawn.resident
2014-12-22 18:59:20,163 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Received a resend of already processed packet #3272, type ParcelPropertiesRequest from 111shawn.resident
2014-12-22 18:59:20,163 DEBUG - OpenSim.Region.ClientStack.LindenUDP.LLUDPServer [LLUDPSERVER]: Receive
[11:15]  Justin Clark-Casey: mic: I know what you're saying, but I'm often surprised to hear new about problems that are actually long standing. It could be increasing use and ppl talking about this is making it more visible
[11:15]  Mic Bowman: and now with latency issues...
[11:15]  Dahlia.Trimble but throttles are on a per client basia arent they?
[11:15]  Mic Bowman: stuff is piling up
[11:15]  Justin Clark-Casey: mic: or, I (or someone else) could have unwittingly inserted a bug
[11:15]  Aine.Caoimhe it was a mix.....vast majority of the people in the region were local
[11:15]  Aine.Caoimhe then there were maybe 4-5 HG visitors
[11:15]  Justin Clark-Casey: I would be surprised if it were throttles as they are per client
[11:15]  Aine.Caoimhe the one who caused the issues was a HG user
[11:15]  BlueWall.Slade did you pin it to a particular visitor?
[11:15]  BlueWall.Slade Oh, ok
[11:15]  Aine.Caoimhe but I've see then exact same thing with a local user too
[11:15]  Mic Bowman: justin... i've been spending quite a bit of time looking at the code
[11:15]  Nebadon Izumi: Diva said she found some interesting things with Inventory requests, though i dont think that is the case here
[11:15]  Mic Bowman: there is a linear search through the resend queue
[11:16]  Justin Clark-Casey: seth: I am still waiting for someone to record a stat trace when these issues happen
[11:16]  Alicia.Raven ive seen this problem going back along time, just dont get people with bad connections come in that oftern
[11:16]  Mic Bowman: if that queue gets big
[11:16]  Mic Bowman: and its being searched with a lock
[11:16]  Justin Clark-Casey: mic: yes, though thinking about it further, I think even a big queue wouldn't delay things much... but that is speculatino
[11:16]  Mic Bowman: then everyone would see crappy performance
[11:16]  Seth.Nygard I've been trying to get some data during one of these events, but so far nothing useful
[11:16]  Dahlia.Trimble yes, its as if a hg person coming in with many assets from a place with a slow or buggy asset server will kill the sim with http requests
[11:16]  Mic Bowman: i think clearly we can making the estimator longer
[11:16]  Aine.Caoimhe exactly, Dahlia
[11:16]  Justin Clark-Casey: the hg asset requests are not throttled in master
[11:16]  Mic Bowman: given the error messages we're seeing
[11:17]  Justin Clark-Casey: are now
[11:17]  Nebadon Izumi: brb someone at my door
[11:17]  Aine.Caoimhe and maybe we're just seeing it so often now because there's so many more people HGing around these days
[11:17]  Dahlia.Trimble so its not udp packets, its http
[11:17]  Mic Bowman: what? hg asset requests are throttled?
[11:17]  BlueWall.Slade In cases like that - would it be better to use microthreadding?
[11:17]  Justin Clark-Casey: mic: yes
[11:17]  Mic Bowman: that's not the adaptive throttle code
[11:17]  Mic Bowman: that adaptive throttles are solely for client connections
[11:17]  Justin Clark-Casey: yes, that's entirely different...
[11:18]  Mic Bowman: hmm...
[11:18]  Justin Clark-Casey: personally, I think it's impossible to say without rerproducing the probelm or getting a stat trace which shows an identifiable issue
[11:18]  Mic Bowman: right... but there seems to be in the mantis entries some combination of low latency connections
[11:18]  Mic Bowman: and hg connections
[11:19]  Aine.Caoimhe I have a feeling it's probably a case where no one thing triggers it...that perhaps it's a case where two borderline things happen to both occur at the same time
[11:19]  Dahlia.Trimble longs for the good old days when network code that used multiple connects used select()...
[11:19]  Mic Bowman: all three entires i've been following mention it
[11:19]  Billy.Bradshaw The grid owner from the party last night said adaptive throttles had been disabled.
[11:19]  Seth.Nygard I tried to replicate the problem with only adding high latency but could not cause the same issue in controlled conditions
[11:19]  Mic Bowman: seth: if you figure it out please let me know
[11:19]  Mic Bowman: i can't reproduce it either
[11:19]  Justin Clark-Casey: seth: how did you add high latency?
[11:20]  Seth.Nygard In our case last night, I know adaptive throttles were enabled
[11:20]  Aine.Caoimhe when it does, general chat suddenly becomes horribly laggy, scripts stop responding, you see serious time dilation (0.6 or worse) and sim fps and physics fps drop from the usual 55 down into the 20's and 30's
[11:20]  Mic Bowman: been trying to figure out how to get a network emulator to add latency
[11:20]  Nebadon Izumi: back
[11:20]  Alicia.Raven i was wondering if firestorms debug settings can simulate lost packets
[11:20]  Justin Clark-Casey: if actually scene fps changes then it's unlikely to be anything to do with throttles
[11:20]  Mic Bowman: seth: please do a show throttles and show queues
[11:20]  BlueWall.Slade Linux TC
[11:20]  Mic Bowman: next time that happens
[11:20]  Mic Bowman: and post to mantis
[11:20]  Mic Bowman: or send to me
[11:20]  Seth.Nygard I am routing through a linux box, as using the traffic control to add latency, with the netem module
[11:20]  Andrew.Hellershanks Aine, the issue nebaon said I had reported to him recently was related to Warp3D. Not sure if if is the same/similar issue to what you have experienced
[11:20]  Mic Bowman: i'll bump up the priority of adding the logging
[11:21]  Justin Clark-Casey: seth: on all udp packets?
[11:21]  Dahlia.Trimble its like the same effect of teleporting to a sim with an empty sim-side asset cache
[11:21]  Seth.Nygard I initially set for all packets, as a wost case
[11:21]  Justin Clark-Casey: seth: how much of a delay?
[11:21]  Aine.Caoimhe it just seems like thread lock of a flooded queue, but none of the debug reports shows anything
[11:21]  Mic Bowman: dahlia... that suggests that there is some resource sharing between client connections and asset requests
[11:21]  Justin Clark-Casey: aine: as I said, someone needs to record a stat trace
[11:21]  Seth.Nygard 500ms +/- 450 with a normal distribution
[11:22]  Mic Bowman: there was a bug in older version of mono where the number of http requests outstanding could shut down a sim
[11:22]  Dahlia.Trimble Mic, yes, I suspect its threadpool related
[11:22]  Seth.Nygard That was to try and replicate what I see from out user with the satellite link
[11:22]  Mic Bowman: threadpool starvation could cause packet handling to be slow... which could push us over the cliff on the resends
[11:23]  Mic Bowman: one thing to look at... again from the mantis entires...
[11:23]  Dahlia.Trimble or cause everything to be slow
[11:23]  Justin Clark-Casey: if someone records stats then tp starvation will be seen
[11:23]  Mic Bowman: its the resend queue size that gets big
[11:23]  Mic Bowman: thats not object updates
[11:23]  Mic Bowman: which are (or at least were) put back into the task queue
[11:23]  Aine.Caoimhe the one person last night managed to do 8k resends in less than the first 5 minutes of being in the sim....everyone else had been there for 60+ minutes and was under 2k total resends
[11:23]  Dahlia.Trimble console is usually spewing http messages
[11:23]  Mic Bowman: so that we resend current state of the object rather than old state
[11:24]  Mic Bowman: one quick fix would be to combine the resend and task queues
[11:24]  Nebadon Izumi: I was able to trigger similar behavior by connecting like 60+ radegast clients to the load tests
[11:24]  Justin Clark-Casey: I hope this isnt' going to be some stabbing in the dark....
[11:24]  Justin Clark-Casey: :)
[11:24]  Mic Bowman: cause the viewer sets the resend queue to some incredibly low number
[11:25]  Nebadon Izumi: my router seems to start floundering around 60 clients
[11:25]  Mic Bowman: justin: it is... but at least this one has some data behind it, look at the places in mantis where we have "show queues"
[11:25]  Mic Bowman: the resend queue is huge
[11:25]  Justin Clark-Casey: which mantises?
[11:25]  Aine.Caoimhe the other interesting thing that Seth discovered the other day is the sim appears to completely ignore the viewer's throttle setting....a viewer set to 250kbs will still get sent at far higher rate...whatever the region server thinks it can spit out
[11:26]  Mic Bowman: let me get it
[11:26]  Aine.Caoimhe not sure if that could be in any way related
[11:26]  Nebadon Izumi: even with the throttle it can burst much higher
[11:26]  Nebadon Izumi: takes a bit for the throttle to kick in
[11:26]  Justin Clark-Casey: seth: what code are you using?
[11:26]  Mic Bowman: the throttles do accommodate bursts by design
[11:26]  Richardus.Raymaker sorry, distracted. what issue with warp3d ?
[11:27]  Aine.Caoimhe stock built from git within the last 2 weeks on both region and grid
[11:27]  Justin Clark-Casey: aine: are you and seth operating the same simulators?
[11:27]  Aine.Caoimhe Seth is running the grid server and some of the region servers, I'm hosting some additional regions of my own
[11:27]  Seth.Nygard The region last night was running code fro, Dec 10
[11:28]  Aine.Caoimhe gut the main ones where we hold parties (or expect high traffic) are all in the same network as the grid server
[11:28]  Mic Bowman: justin: #7308
[11:28]  Seth.Nygard Last night the region server is in a different data center than the grid server
[11:28]  Mic Bowman: though it looks like byes unacked
[11:28]  Mic Bowman: so it might still be task queue
[11:28]  Justin Clark-Casey: mic: that screenshot shows on 2 packets queued for resend, though many resent
[11:28]  Mic Bowman: also... it looks like acks are coming back
[11:29]  Mic Bowman: given the number of packets in
[11:29]  Mic Bowman: which suggests that we aren't waiting long enough for the acks
[11:29]  Mic Bowman: either thread problem (not handling them fast enough)
[11:29]  Mic Bowman: or the RTT is broken
[11:29]  Justin Clark-Casey: that's possible. though it seemd to me any more than a few seconds isn't worth acking anyway?
[11:29]  Andrew.Hellershanks Seth, I saw issues in a grid when the servers were in two different locations.
[11:30]  Mic Bowman: agreed but 13K unacked means that the window is pretty large
[11:30]  Mic Bowman: or was
[11:30]  Justin Clark-Casey: that is bytes unacked
[11:30]  Justin Clark-Casey: not packets
[11:30]  Mic Bowman: correct
[11:30]  Justin Clark-Casey: 13k doesn't seeem that much
[11:30]  Mic Bowman: thats still 15+ packets
[11:30]  Mic Bowman: with multiple updates per packet
[11:31]  Justin Clark-Casey: really, this shows up much more in the resent number
[11:31]  Justin Clark-Casey: pCampbot running many bots triggers the same thing, because pCampbot doesn't process received packets quickly enough, I think
[11:31]  Mic Bowman: its both
[11:32]  Mic Bowman: unacked are packets that have been put on the wire
[11:32]  Justin Clark-Casey: we didn't suffer such issue with pCampbot stuff, which is massively latent
[11:32]  Justin Clark-Casey: though we were never running for more than 2 hours
[11:32]  Mic Bowman: not disagreeing
[11:32]  Mic Bowman: just hard to imagine that after five years of running this code, this is the first time we've had problems
[11:32]  Justin Clark-Casey: tbh, it doesn't surprise me
[11:32]  Mic Bowman: the throttle code is about that old
[11:33]  Mic Bowman: we didn't see it at last years OSCC did we?
[11:33]  Justin Clark-Casey: many times I have seen problems which were new to me but had existed for years, just nobody talked about them
[11:33]  Mic Bowman: 2013
[11:33]  Seth.Nygard I do have a wireshark capture of one night we was this issue, but not the corresponding stats log.
[11:33]  Mic Bowman: seth: can you post that to one of the mantis entires (assuming that there is nothing confidential in it)
[11:33]  Justin Clark-Casey: mic: last year's oscc it's honestly hard to say. The numbers were pretty low and everyone was sitting
[11:34]  Justin Clark-Casey: seth: why didn't you capture thre stats log?
[11:34]  Mic Bowman: i'll just say that given the severity... i'm surprised it has been brought up
[11:34]  Seth.Nygard It is a pretty large capture file. I can send it anyone that wants to look at it
[11:34]  Mic Bowman: and the consistency with which its happening
[11:34]  Seth.Nygard That was before we mentioned the need for the stats log.
[11:35]  Justin Clark-Casey: seth: ok, because with the stats log I can produce a set of graphs
[11:35]  Seth.Nygard nods
[11:35]  Justin Clark-Casey: and it will be very obvious if there is tp exhaustion, for instance
[11:35]  Seth.Nygard I have some stats logs, but nothing so far that coincides with a large number of issues
[11:35]  Aine.Caoimhe the number of HG travelers it up pretty dramatically recently, if that's somehow a contributing's certainly something I've been experiencing inworld far more in the last 3-4 months than ever before
[11:36]  Nebadon Izumi: could be Aine
[11:36]  Justin Clark-Casey: aine: yes, I wouldn't be surprised if that isn't playing a role
[11:36]  Justin Clark-Casey: I believe it did for oscc, for instance
[11:36]  Aine.Caoimhe nods
[11:36]  Mic Bowman: HG shouldn't affect the adaptive throttles unless its triggering something
[11:36]  Justin Clark-Casey: yeah, I actually doubt this is a throttles problem
[11:37]  Mic Bowman: something is triggering a fatal behavior in the throttles
[11:37]  Aine.Caoimhe it seems like it needs a perfect storm of conditions to trigger it, but those are being achieved a lot more often recently
[11:37]  Seth.Nygard I have tried regions with and without adaptive throttles and see the same behaviour when this happens
[11:37]  Mic Bowman: strongly agree aine!
[11:37]  Aine.Caoimhe which is what makes me thing HG is at least an element of it
[11:37]  Justin Clark-Casey: mic: If it were throttles, I would not expect the scene fps to fall
[11:37]  Mic Bowman: correct
[11:37]  Aine.Caoimhe mesh might be another since a lot more people are wearing mesh than used to be the case
[11:37]  Justin Clark-Casey: which is what some are reporting
[11:38]  Justin Clark-Casey: and another thing that would show up in stats traces....
[11:38]  Nebadon Izumi: we saw something ineresting on MOSES grid
[11:38]  Mic Bowman: unless we are spending so much tim processing resends that we aren't getting other useful work done
[11:38]  Nebadon Izumi: which doesnt quite apply but maybe
[11:38]  Justin Clark-Casey: mic: that only happens on one thread
[11:38]  Nebadon Izumi: since MOSES isnt HG enabled, we are having to move over content via IAR and OAR
[11:38]  BlueWall.Slade ++JC, can't argue with good instrumentation
[11:38]  Nebadon Izumi: but these IAR and OAR objects tend to retain the last owner UUID
[11:38]  Nebadon Izumi: which somtimes puts their grid into a looping lookup request
[11:38]  Nebadon Izumi: that it cant ever satisfy
[11:39]  Justin Clark-Casey: neb: how do you know?
[11:39]  Aine.Caoimhe yes, I've see than one too (and logged a mantis on it)
[11:39]  Marcus.Llewellyn I've seen what Neb is describing on my little grid, after importing my OSgrid IAR.
[11:39]  Nebadon Izumi: been looking for perforamce issues with Doug
[11:39]  Nebadon Izumi: we did a small load test yesterday
[11:39]  Nebadon Izumi: and that was one of the things taht popped out at him
[11:39]  Justin Clark-Casey: neb: how does he know?
[11:39]  Mic Bowman: so... i'll add more logging code justin... i want to see what percentage of packets are never acked vs percentage that are acked late
[11:39]  Nebadon Izumi: i wasnt watching consoles, mostly relaying what he said
[11:39]  Justin Clark-Casey: neb: really difficult to trust second hand info
[11:40]  Justin Clark-Casey: mic: ok, it would be great if you could add that to existing stats data....
[11:40]  Nebadon Izumi: i suspect in that case anyway justin its more of a simiangrid issue
[11:40]  Aine.Caoimhe @Neb:
[11:40]  Mic Bowman: might need some pointers for how to do that
[11:40]  Mic Bowman: but sure
[11:40]  Mic Bowman: sounds like the right thing to do
[11:41]  Marcus.Llewellyn I get a huge a mount of spew like the following:
[11:41]  Marcus.Llewellyn 2014-10-16 12:52:49,615 DEBUG - OpenSim.Region.CoreModules.Framework.UserManagement.UserManagementModule [USER MANAGEMENT MODULE]: No grid user found for 192523fa-57e3-274e-48c7-a6a0adaeaa81
[11:41]  Justin Clark-Casey: sure. At the top of LLUDPServer it shows how a bunch are registered. But very happy to talk about it more when necessary
[11:41]  Justin Clark-Casey: mic: if it's in that framework it can then also be recorded, etc.
[11:41]  Seth.Nygard Other than good stats and logs, what other data can I collect that is helpful to track these issues down?
[11:42]  Mic Bowman: thanks justin... i'll start there & send email if i have questions
[11:42]  Aine.Caoimhe console login to Justin so he can watch it live :p
[11:42]  Justin Clark-Casey: seth: For me, it's all about the stats record at this stage. That gives actual insight into the state of the system.
[11:42]  Justin Clark-Casey: I used this a lot when looking into oscc problems
[11:42]  Seth.Nygard nods
[11:42]  Andrew.Hellershanks Marcus, I have seen a couple of instances where a record in the user accounts table would go missing.
[11:43]  Marcus.Llewellyn These are users that don't exist on my grid. They're likely OSgrid UUIDs.
[11:43]  Marcus.Llewellyn I have a log section where that goes on for around 120 lines.
[11:43]  Justin Clark-Casey: this user management stuff is getting complicated
[11:44]  BlueWall.Slade It should give a good hones try to find them, then give up.
[11:44]  BlueWall.Slade honest*
[11:44]  Justin Clark-Casey: bluewall: i thought it did cache negatives
[11:44]  Marcus.Llewellyn It takes about two minutes, and bogs the sim while it's happening.
[11:44]  Justin Clark-Casey: but doesn't sound like it
[11:45]  Mic Bowman: whats the timeout on the cache
[11:45]  BlueWall.Slade It seems that some conditions can make it go for a long time.
[11:45]  Mic Bowman: and do we still do those lookups for permission checks?
[11:45]  Aine.Caoimhe maybe that's just long enough for it to also stack up a queue of other stuff that has issue if it doesn't get processed rapidly enough
[11:45]  Mic Bowman: at one point every login tried to retrieve user ids for all object owners for permission checks
[11:46]  Justin Clark-Casey: mic: looking right now at the UMM module, the m_UserCache is just a dictionary with no timeout
[11:46]  Marcus.Llewellyn My asset cache has a 48 hour timeout.
[11:46]  Dahlia.Trimble maybe they just time out because the asset server cant fulfill the requests at the rate the sim asks for them?
[11:46]  BlueWall.Slade I put a manual command to invalidate it last week
[11:46]  Justin Clark-Casey: that thing is a bit of a mess
[11:46]  Mic Bowman: that's asset cache... what about user cache?
[11:46]  Aine.Caoimhe @Dahlia: or is throttling them
[11:46]  BlueWall.Slade In some things, needt top kick a sim over
[11:46]  Marcus.Llewellyn I'm uncertain where to find that.
[11:47]  Justin Clark-Casey: mic: I'm not sure what you're asking
[11:47]  Mic Bowman: nevermind
[11:47]  Mic Bowman: i just know that we got killed on group lookups for permissions
[11:48]  Justin Clark-Casey: There's a lot of awful ad-hoc stuff
[11:48]  Mic Bowman: had to add a simian specific cache for non-existant users and groups
[11:48]  Marcus.Llewellyn The messages I'm talking about seem to coincide with inventory loading on the client.
[11:49]  Aine.Caoimhe yeah...inventory load is another huge issue....impossible to log in to any region not hosted on the same network as the grid server without inventory fetch hanging infinitely and also brining the entire region to its knees
[11:49]  Justin Clark-Casey: would also be good to see a stats trace on current behaviuor of that too
[11:49]  BlueWall.Slade si that on other networks besides OSGrid?
[11:50]  Justin Clark-Casey: from point of login through to where such issues manifest
[11:50]  Aine.Caoimhe yes....happens on ours
[11:50]  Aine.Caoimhe I can't log in to any region I host
[11:50]  Marcus.Llewellyn On my little one man grid as well.
[11:50]  Seth.Nygard I've seen it on several grids
[11:50]  Justin Clark-Casey: marcus: are you able to generate a stats trace for that?
[11:50]  BlueWall.Slade I don't see it here, but I'm all in one network.
[11:50]  Seth.Nygard I can generate stats for that as well
[11:51]  Justin Clark-Casey: ok, thanks
[11:51]  Aine.Caoimhe (I can, but it will then sit there generating 1000+ HTTP fetches per minute until I log our again and then takes up to an hour to finally process the queue even after I've logged out
[11:51]  Marcus.Llewellyn I have to admit I've never done that. I'm happy to do it, though.
[11:51]  Aine.Caoimhe Bluewall and I spent a Sunday doing tests on that in the spring
[11:51]  Marcus.Llewellyn I'm pretty sure logging in with a clean viewer cache repeats the issue for me.
[11:51]  Justin Clark-Casey: marcus:
[11:51]  Justin Clark-Casey: is is the case that whatever else, we should remove pending requests if the user leaves the region
[11:52]  Aine.Caoimhe I don't even need to clear cache...happen on every single login
[11:52]  Marcus.Llewellyn Ah, didn't know about that functionality. Thanks. Will do that. :)
[11:52]  BlueWall.Slade Maaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
[11:52]  Nebadon Izumi: lol
[11:52]  BlueWall.Slade sorry about that
[11:53]  Dahlia.Trimble blames the cat for sitting on the keyboard
[11:53]  Marcus.Llewellyn Is there an extant mantis for this issue?
[11:53]  BlueWall.Slade Marcus - you grid. Is it on several servers?
[11:53]  Andrew.Hellershanks Hey, it wasn't my cat. He doesn't do that
[11:53]  Justin Clark-Casey: marcus; the inventory one?
[11:54]  Marcus.Llewellyn My grid has two machine right now. Robust/Mysql is on one. The simulators are on another.
[11:54]  Marcus.Llewellyn nods to JCC
[11:54]  BlueWall.Slade ok, in-house?
[11:54]  Aine.Caoimhe
[11:54]  Justin Clark-Casey: marcus:
[11:54]  Marcus.Llewellyn Yes, all on the same LAN.
[11:54]  BlueWall.Slade ok, thanks
[11:54]  Marcus.Llewellyn Thanks, jcc. :)
[11:54]  Andrew.Hellershanks I've heard of another odd cache(?) related issue as of a few days ago but it seems to involve Warp3D
[11:54]  Justin Clark-Casey: warp3d is a memory leaking mess
[11:55]  Justin Clark-Casey: internally
[11:55]  Nebadon Izumi: this is different Justin
[11:55]  Andrew.Hellershanks yeah, I've heard that. The person using it has it set to run every 6 hours.
[11:55]  Dahlia.Trimble does warp3d leak on windows also?
[11:55]  Nebadon Izumi: this seems to be causing sim lag for him when someone pulls up the map
[11:55]  Nebadon Izumi: which is odd, and it doesnt happen when warp3d is off
[11:55]  Justin Clark-Casey: dahlia: don't know
[11:55]  Nebadon Izumi: I don't get that at all
[11:55]  BlueWall.Slade maybe the textures are more detailed/larger?
[11:56]  Marcus.Llewellyn was pondering using the dispatcher with an externalised version of Warp3D. :)
[11:56]  Nebadon Izumi: that seems unlikely to me
[11:56]  Andrew.Hellershanks Run grid using Warp3D. Login to grid. First time user calls up map, the instance/region locks upfor about 60 seconds. Console doesn't show any errors other than missed heartbeat. Next person in region to pull up map for first time after login will cause same issue.
[11:56]  Nebadon Izumi: we would be seeing texture fetching causing more problems if that were the case I would think
[11:56]  Dahlia.Trimble Ive seen a few cases where code that creates many small objects leaks on mono but not windows
[11:56]  BlueWall.Slade Those tiles are served from Robust
[11:56]  Justin Clark-Casey: dahlia: this is a leak due to failure to dispose of graphics objects, I believe
[11:56]  Dahlia.Trimble as if they dont free on mono
[11:56]  Nebadon Izumi: right bringing up the map shouldnt invoke warp3d in any way should it?
[11:57]  Nebadon Izumi: very strange
[11:57]  BlueWall.Slade Warp3d will send the tiles to Robust when they are created. After that, it's nothing to do with the regin
[11:57]  Marcus.Llewellyn GDI+ on Mono has always been... hinky.
[11:57]  Justin Clark-Casey: dahlia: ordinary tile generator was leaking the same way until I fixed it to properly clean up afterwards
[11:57]  Justin Clark-Casey: if you have IDiposable objects you need to dispose of them
[11:57]  Andrew.Hellershanks Robust, and I would have expected them map tiles to cached in Robust. When using standard map tile generator no lockup of region/instance is being seen so it appears related specifically to Warp3D
[11:57]  Justin Clark-Casey: it's not GDI's fault, it's bad programming
[11:58]  Richardus.Raymaker Not that i know. map use maptiles and the are sitting on the rovbust disk.
[11:58]  Nebadon Izumi: lately I have been generating my own tiles with the viewer a screenshot and photoshop
[11:58]  Nebadon Izumi: and hard coding tiles
[11:58]  Nebadon Izumi: looks so much nicer and less stress on the simulator
[11:58]  Justin Clark-Casey: labour intensive :)
[11:58]  Richardus.Raymaker so.. how can warp3d screw things when you open map ?
[11:59]  BlueWall.Slade It wouldn't hurt to run Warp3d on startup.
[11:59]  Nebadon Izumi: I don't know Richardus, that doest make much sense
[11:59]  Andrew.Hellershanks Nebadon, I'd like to know how you take the pictures. Might be possible to automate the post capture proccess
[11:59]  Justin Clark-Casey: you guys can speculate all you want, it's meaningless until someone looks at the code
[11:59]  Dahlia.Trimble justin, ya I guess failure to dispose (not using "using") could do it also
[11:59]  Richardus.Raymaker Bluewall, i only run it at startup
[11:59]  Nebadon Izumi: i sit on a cube
[11:59]  Nebadon Izumi: and move it to certain coordinates
[11:59]  Andrew.Hellershanks Warp3D does run on reegion startup to generate maps.
[11:59]  BlueWall.Slade Nebadon - Mesh terrains make bad map tiles, lol
[11:59]  Nebadon Izumi: then go into mouseloook
[11:59]  Richardus.Raymaker Hands justin a plate with spaghetti :)
[11:59]  Marcus.Llewellyn I use so much mesh now... my maptiles are just big blocks
[11:59]  Nebadon Izumi: yes but with mesh terrain
[11:59]  Justin Clark-Casey: richardus: I'm just being grouchy :)
[11:59]  Nebadon Izumi: its super easy to take your texture
[11:59]  Nebadon Izumi: on the mesh and apply it as a map tile hard coded
[12:00]  BlueWall.Slade Nebadon, you could probably script a camera to do that
[12:00]  Nebadon Izumi: yea probably
[12:00]  Marcus.Llewellyn +1 Bluewall
[12:00]  Aine.Caoimhe render it in Blender
[12:00]  Nebadon Izumi: this is how LL does it
[12:00]  Nebadon Izumi: they made a custom viewer
[12:00]  Andrew.Hellershanks Nebadon, send me the coords sometime and I'll try it out.
[12:00]  Justin Clark-Casey: richardus: but yeah, the reason I haven't simply fixed warp3d to dispose properly is because it's spaghetti like
[12:00]  Nebadon Izumi: that takes snapshots and crawls the grid
[12:00]  Justin Clark-Casey: the objects are passed all around the place so it's not easy to track and dispose of them consistently
[12:00]  Richardus.Raymaker justin :)
[12:00]  Justin Clark-Casey: though even best effort would probably be better than the situation now
[12:00]  BlueWall.Slade LL updates the map tiles every few months
[12:01]  Nebadon Izumi: I think it was like 128/128/300
[12:01]  Nebadon Izumi: or 128/128/320
[12:01]  Nebadon Izumi: something like that id have to look on other grid
[12:01]  Andrew.Hellershanks justin, i've tried looking at warp3d code but I don't feel I know enough about object garbage collection in C# to find the source of the leak, nor have I found a way to test to see if the code is leaking.
[12:01]  Nebadon Izumi: most of the tiles on this grid are all hard coded screenshots i made
[12:02]  Justin Clark-Casey: andrew: I't simply because the IDisposable graphics objects are never Disposed, I expect
[12:02]  BlueWall.Slade Nebadon, I'll send you a tool to help script a camera
[12:02]  Nebadon Izumi: search for Keynote on map
[12:02]  Andrew.Hellershanks Nebadon, ok. Did you have to rescale after in PS?
[12:02]  Nebadon Izumi: yes
[12:02]  Justin Clark-Casey: that's certainly the case, but I couldn't definitively say that's the whole story until at least that is fixed
[12:02]  Andrew.Hellershanks Nebadon, ok
[12:02]  Nebadon Izumi: it needs to be the # of pixels to # of meters
[12:02]  Dahlia.Trimble should just rip the warp and oar code out and make a program that builds map tiles out of an oar file
[12:02]  Andrew.Hellershanks Justin, I may have another look. I can't test code that easily right now with my main desktop being dead.
[12:02]  BlueWall.Slade Mic, thanks for the Dispatcher docs. They look good.
[12:02]  Andrew.Hellershanks Might as well spend some time digging through code.
[12:03]  Marcus.Llewellyn If an automatced crawler is desired, Radegast's 3D view is prolly good enough for it, and that client is easy to modify.
[12:03]  Justin Clark-Casey: mic: did you want to try and informal load test now?
[12:03]  Justin Clark-Casey: seeing as we are at the hour
[12:03]  Mic Bowman: bluewall: i've moved everything to a new git repository
[12:03]  Andrew.Hellershanks One other quick question before we all go "poof" for the week
[12:03]  Mic Bowman:
[12:03]  Nebadon Izumi: Marcus, but then you don't get shadows and windlight
[12:03]  Nebadon Izumi: it would look like poo in Radegast
[12:03]  BlueWall.Slade yep, that's what I'm loking at.
[12:04]  Mic Bowman: ok
[12:04]  Marcus.Llewellyn True, but from a couble hundred meters up, that doesn't realy make a dramatic difference anyway. And it still would beat the map generators we got now.
[12:04]  Andrew.Hellershanks Is there extra call/processing to handle things like ${Const|PrivatePort} in ini settings? Seems one INI doesn't handle some settings.
[12:04]  Dahlia.Trimble do you get shadows and windlight in warp?
[12:04]  Mic Bowman: justin: anything we can get is helpful
[12:04]  Mic Bowman: though i would prefer to get the stats added
[12:04]  BlueWall.Slade No Andrew
[12:04]  Andrew.Hellershanks Either XBakes or Profiles doesn't like that in the URL
[12:04]  Mic Bowman: can we shoot for next tuesday?
[12:04]  Justin Clark-Casey: what do you think Neb
[12:04]  Justin Clark-Casey: ?
[12:04]  Nebadon Izumi: sure
[12:05]  Nebadon Izumi: I am available
[12:05]  Mic Bowman: trying to finish dispatcher work before moving to throttles
[12:05]  BlueWall.Slade Andrew, I use it throughout.
[12:05]  Mic Bowman: (given your comment about not being able to work on it for awhile)
[12:05]  Nebadon Izumi: lol
[12:05]  Nebadon Izumi: who's snowballing me
[12:05]  Aine.Caoimhe have a great week and a merry Christmas, everyone :)
[12:05]  Justin Clark-Casey: merry xmas Aine
[12:06]  BlueWall.Slade Thanks Aine, same you you.
[12:06]  Marcus.Llewellyn B'bye Aine :)
[12:06]  Mic Bowman: merry christas!
[12:06]  Seth.Nygard yes, Merry Christmas to everyone
[12:06]  Dahlia.Trimble bye Aine :)
[12:06]  Justin Clark-Casey: mic, neb: ok, next week then.
[12:06]  Dahlia.Trimble Merry Christmas all :D
[12:06]  Nebadon Izumi: cool, sounds like a plan
[12:06]  Aine.Caoimhe waves and *poofs*
[12:06]  Richardus.Raymaker merry christmas
[12:06]  Mic Bowman: sonds good justin
[12:06]  BlueWall.Slade Merry Christmas!
[12:06]  Andrew.Hellershanks Marcus, I wouldn't try digging through use of Radegast to look at having it crawl to do map tiles as viewer code is C++ and I don't do C++
[12:06]  Nebadon Izumi: ya Merry Christmas everyone, thanks for coming, great turn out today
[12:06]  Nebadon Izumi: awesome meeting
[12:06]  Andrew.Hellershanks Merry Christmas everyone.
[12:06]  Marcus.Llewellyn Radegst is C#.
[12:06]  Justin Clark-Casey: I'm pretty sure radegast is c#
[12:06]  Aine.Caoimhe lol...nope, can't tp back home
[12:07]  Andrew.Hellershanks Oh, ok. I tend to assume most viewer related stuff is C++ as they usually are.
[12:07]  Aine.Caoimhe why does it see me on my internal IP
[12:07]  Andrew.Hellershanks Most are based on common code base coming from SL
[12:07]  BlueWall.Slade haha what is that?
[12:07]  Marcus.Llewellyn I tend to regard Radegast at the real documentation for how to use libOMV in a client. Hehe.
[12:07]  Andrew.Hellershanks :)
[12:07]  Richardus.Raymaker homework for nebadon. better chairs. or a parachute
[12:07]  Aine.Caoimhe 15:06:01 - [SCENE PRESENCE]: Completing movement of Aine Caoimhe into region Aine Workshop in position <128.025, 127.891, 25.419>
15:06:01 - [SCENE]: User Client Verification for Aine Caoimhe in Aine Workshop returned false
15:06:01 - [CLIENT]: Close has been called for Aine Caoimhe attached to scene Aine Workshop
15:06:01 - [SCENE]: Removing child agent Aine Caoimhe 12a7488f-d08c-448e-93ff-6ae67500f709 from Aine Workshop
15:06:01 - [CAPS]: Remove caps for agent 12a7488f-d08c-448e-93ff-6ae67500f709 in region Aine Workshop
15:06:01 - [CAPS]: Unauthorized CAPS client 12a7488f-d08c-448e-93ff-6ae67500f709 from
[12:07]  Nebadon Izumi: Redagest Rendering is cool
[12:07]  Nebadon Izumi: but it has issues
[12:07]  Billy.Bradshaw anie, may need to clear hyperlinks
[12:08]  Richardus.Raymaker some chairs are a trap !
[12:08]  Justin Clark-Casey: aine: that sometimes means the user agent service on this installation could not be contacted
[12:08]  Seth.Nygard take care everyone, have a great day
[12:08]  Andrew.Hellershanks I'll have the user with the warp3d issue check the Roust console when the reigon locks up
[12:08]  Richardus.Raymaker ok. lets see if we can TP home. bye all
[12:08]  Aine.Caoimhe kk...guess I log out and back in again then
[12:08]  Justin Clark-Casey: aine: one moment
[12:08]  Seth.Nygard waves and departs
[12:08]  Aine.Caoimhe kk
[12:08]  Seth.Nygard is Online
[12:08]  Andrew.Hellershanks oh, yeah. I wonder if I can get back to where I came from
[12:09]  Marcus.Llewellyn I ain't claiming Radegast will give you the fidelity the LL viewer can. It sure won't. But it fits into the "good enough" category.
[12:09]  Richardus.Raymaker dont lean back nebadon. theres a snowball
[12:09]  Dahlia.Trimble bye all :) *ctrl-shift-h*
[12:09]  Nebadon Izumi: im actually in my home region :)
[12:09]  Marcus.Llewellyn B'bye Dahlia
[12:09]  Aine.Caoimhe shouts: my initial TP contact was this: 15:05:59 - [SCENE]: Region Aine Workshop told of incoming child agent Aine Caoimhe 12a7488f-d08c-448e-93ff-6ae67500f709 (circuit code 1108777924, IP, viewe
r Firestorm-Releasex64, teleportflags (ViaHome, ViaHGLogin), position <127.9989, 127.9999, 25.96604>. From region Staff Zone 1 (75a5b0cf-9d1d-4f30-8dcf-b25de6
eeae64) @
[12:09]  Mic Bowman: bye all
[12:09]  Dahlia.Trimble is Online
[12:09]  Nebadon Izumi: see you Mic
[12:09]  Marcus.Llewellyn Cya Mic
[12:09]  Aine.Caoimhe so in that one I have the correct IP
[12:09]  Mic Bowman is Offline
[12:09]  Aine.Caoimhe then when it does the verification is has somehow changed to my log IP
[12:10]  Justin Clark-Casey: aine: tihs is going back to your home grid?
[12:10]  Aine.Caoimhe this is going back to a home-hosted region on our grid, yes
[12:10]  Aine.Caoimhe ie the region is one I host, not that the grid hosts
[12:10]  Justin Clark-Casey: aine: the failure point is when your simulator attempts to contact your own user agent service for verification and fails for some reason
[12:10]  Justin Clark-Casey: I would initially suspect some bug in opensim tbh
[12:11]  Justin Clark-Casey:
[12:11]  Justin Clark-Casey: a failure on step 6.5
[12:11]  BlueWall.Slade Take care everyone. See you all next wek.
[12:11]  BlueWall.Slade is Online
[12:11]  Aine.Caoimhe yeps
[12:11]  Aine.Caoimhe that looks like what's happening
[12:12]  Aine.Caoimhe so when it tries to do that, it's seeing my LAN IP not my external facing IP
[12:12]  Aine.Caoimhe or actually it's seeing my router's LAN IP
[12:12]  Justin Clark-Casey: aine: it might be something like that. I need to put in yet more debugging info, though the current structure of code makes that hard
[12:13]  Aine.Caoimhe I think I can tp back to a grid-hosted region though....just not one of my own
[12:13]  Aine.Caoimhe much like I can't log in to any region I host
[12:13]  Dahlia.Trimble is Offline
[12:13]  Aine.Caoimhe (without fetch bug killing it)
[12:14]  Aine.Caoimhe anyway...such is life...I'll just always go via a grid-hosted region
[12:14]  Justin Clark-Casey: aine: could you try the tp back again?
[12:14]  Aine.Caoimhe to my own one?
[12:14]  Alicia.Raven gotta go, bye everyone
[12:14]  Justin Clark-Casey: yes please
[12:14]  Justin Clark-Casey: bye alicia
[12:14]  Alicia.Raven merry christmas all
[12:14]  Aine.Caoimhe goes
[12:15]  Justin Clark-Casey:
[12:15]  Justin Clark-Casey: is that familiar at all?
[12:15]  Justin Clark-Casey:
[12:15]  Justin Clark-Casey:
[12:15]  Aine.Caoimhe um....trying to think if it might be our grid's
[12:16]  Aine.Caoimhe my outward facing one is the
[12:16]  Marcus.Llewellyn I"m gonna scoot.
[12:16]  Marcus.Llewellyn smiles and waves. :)
[12:16]  Aine.Caoimhe bye Marcus
[12:17]  Justin Clark-Casey: yeah, something is going on but I think the ultimate failure may be when your own simulator contacts your region's user agent service to verify that the transport is ok
[12:18]  Justin Clark-Casey: I can see that IP when this region contacts your region to tell it that the agent is coming
[12:18]  Aine.Caoimhe somewhere in there it's switching over to using my router's LAN IP instead of my outward facing one, then of course that is a mismatch to the original IP the tp request came from
[12:19]  Justin Clark-Casey: actually, from the trace you IM'd it looks like that might not be getting through
[12:19]  Seth.Nygard is Offline
[12:19]  Justin Clark-Casey: aine: I dunno. I would say add a mantis but you know how many are currently on the pile
[12:19]  Aine.Caoimhe I know
[12:19]  Justin Clark-Casey: I am interested in making it easier to diagnose such problems in the long run
[12:20]  Seth.Nygard is Online
[12:20]  Justin Clark-Casey: or fix them if they are opensim bugs
[12:20]  Aine.Caoimhe and it's not the end of the world which is why I haven't made a mantis on it... i know you already have too much on your plate
[12:20]  Justin Clark-Casey: ok, let's keep it in mind though. Do feel free to raise sometihng if you want. Might be good just to keep track
[12:20]  Aine.Caoimhe the big ones for me at the animation bug, the inventory fetch bug, and the killing region bug since they affect everyone
[12:21]  Justin Clark-Casey: yes. I still need to look at the anim bug, my time got caught up with some other issue. The inventory fetch also needs attention
[12:21]  Justin Clark-Casey: ok
[12:21]  Aine.Caoimhe that one-user killing it for everyone is the worst killer
[12:21]  Justin Clark-Casey: yes, hoepfully we can get some progress on that soon
[12:22]  Aine.Caoimhe okies....have a great Christmas, Justin....see you next week :)
[12:22]  Justin Clark-Casey: ok, I'm going to skeddale. Bye for now
[12:22]  Justin Clark-Casey: hope you have a great xmas too Aine
Personal tools
About This Wiki