Chat log from the meeting on 2014-08-12

 [11:02] Sasha Nae: sorry If I'm interupting [11:02] Freaky.Tech @freakytech.spdns.de:8002 is Online [11:02] Andrew Hellershanks is Online [11:02] Nebadon.Izumi @cc.opensimulator.org:8002 is Online [11:03] Shez Oyen: not at all [11:03] Andrew Hellershanks: I don't think the meeting has started yet, Sasha. All are welcome to attend. [11:03] Richardus Raymaker: hi shez [11:03] Shez Oyen: Hi Rich :) [11:03]  Justin Clark-Casey: hi folks [11:03]  Sasha Nae: what is the meeting about? [11:03]  Andrew Hellershanks: Hey, justin [11:03]  Nebadon Izumi: this is the Weekly OpenSimulator Developers office hours meeting [11:03]  Shez Oyen: Olla Justin [11:03]  Mata Hari: hi Justin [11:03]  Richardus Raymaker: hi justin [11:04]  Sasha Nae: Heya Dan Banner [11:04]  dan banner: hello [11:04]  Andrew Hellershanks: I filed two mantis reports this week. [11:04]  Shez Oyen: Sasha the meeting is about development of Opensim/osgrid [11:04]  Kayaker Magic: Yay! [11:04]  Sasha Nae: aaah [11:04]  Sasha Nae: okay [11:04]  BlueWall Slade is Online [11:05]  Andrew Hellershanks: Justin, Can I push the changes to llEmail? [11:05]  Simulator Version v0.5 shouts: OpenSim 0.8.1 Dev          8738445: 2014-08-09 00:52:51 +0100 (Unix/Mono) [11:05]  Justin Clark-Casey: andrew: did you make further changes to that patch on mantis@ [11:05] Justin Clark-Casey: ? [11:06] Andrew Hellershanks: Justin, yes. I added an extra check to see if the destination was local or off-world. If local, the address is used as provided. My previous version would have affect object to object email. [11:06] Sasha Nae: sory [11:06] Shez Oyen: lol [11:06] Justin Clark-Casey: andrew: ok cool. I say go for it [11:06] Sasha Nae whispers: my avatar is lagging AGAIN! [11:07] Andrew Hellershanks: justin, ok. I'll do that later today. [11:07] Andrew Hellershanks: Does anyone have any sample code to test object to object email? [11:07] Richardus Raymaker: Sasha, check your draw distance first [11:07] Richardus Raymaker: http://wiki.secondlife.com/wiki/Llemail [11:07] Shez Oyen: Hi Robert [11:07] Nebadon Izumi: Andrew i dont think you can do object to object email [11:07] Nebadon Izumi: without some kind of pop3 module [11:07] Robert Adams: hello all [11:08] Justin Clark-Casey: hi robert [11:08] Richardus Raymaker: hi robert [11:08] Justin Clark-Casey: nebdadon: just fyi, changing maxconn didn't make any difference to my issue so I think we should leave it as it is now on oscc [11:08] Andrew Hellershanks: Nebadon, I wanted to test that. I haven't dug deep enough in to the internal mail system about that. [11:08] Nebadon Izumi: ok cool [11:08] Nebadon Izumi: ya without pop3 im not sure you can do object to object email [11:09] BlueWall Slade: object to object email only works within the same simulator [11:09] Nebadon Izumi: either pop3 service or some kind of custom module that acts as a delivery service [11:09] Robert Adams: what tangled threads we weave [11:09] Sasha Nae: OSgrid is a ver awesome place to be...seing that everything is free and all [11:09] Andrew Hellershanks: I'll have to look closely at how llGetNextMail (or whatever it is called) gets mail. [11:09] Justin Clark-Casey: I did observe a curious effect where outoing udp is very peaky but after 23 bots out of 100 timed out, the reamining 77 bots had vastly lower and more stable outgoing [11:09] Justin Clark-Casey: maybe I'#m hitting some capacity limit still on my local system [11:10] Nebadon Izumi: hmm could be [11:11]  Robert Adams: queues on top of caches on top of queues... all of which have their own limits [11:12] Justin Clark-Casey: yes, it's a bit thicket of stuff [11:12] Sasha Nae: I want to go to lf grid [11:12] Justin Clark-Casey: not to mention that c# http server which has had custom things done to it.... [11:12] Andrew Hellershanks: Glad I don't dig in to all that stuff. :) [11:12] Robert Adams: OS's http server has always been a weak spot [11:12]  Justin Clark-Casey nods [11:13]  Robert Adams: one reason we built Simian -- to use a real web server [11:13]  Andrew Hellershanks: Could one of the other HTTP server projects been incorporated? [11:13]  Justin Clark-Casey: seems there still aren't good c# webserver implementations, at leaston a casual look yesterday [11:13]  Justin Clark-Casey: robert: yeah, real webservers are definitely the way to go for services [11:13]  Justin Clark-Casey: but still need to deal with an embedded one for the http capabilities in the simulator [11:14]  Justin Clark-Casey: andrew: there still don't seem to be any good ones in c# [11:14]  Justin Clark-Casey: incorporating someting from native code would bring hcallenges of it's own. though we do it for physics after all... [11:14]  Andrew Hellershanks: oh, right. Need it in C# [11:14]  Mata Hari: how extensive an upheaval would it be to pull some of the proxying from the region and let viewer do stuff directly with the assets or inventory servers? [11:14] Robert Adams: since we need only a subset of web-serverness, building our own is an option [11:14] Nebadon Izumi: I thought there was a mono C# web server that supposedly out performed apache [11:15] Justin Clark-Casey: mata: asset stuff can already be done. Inventory is a bit more of a challenge I think because of the general need for security [11:15] Justin Clark-Casey: but somebody may have done that [11:15] Mata Hari nods [11:15] Justin Clark-Casey: robert: true - I always try and avoid that approach though - yet another thing to maintain [11:16] Mata Hari: it seems to be during that point that the inventory fetch gets locked.....Bluewall and I spent a chunk of Sunday data-gathering [11:16] Justin Clark-Casey: it's easy for somebody to make great performance claims for their stuff - I like to see it backed up with numbers :) [11:16]  Justin Clark-Casey: mata: it depends where the problem is happening, there are unfortunately many points of failure [11:16]  Robert Adams: very true -- then one discovers that something is realy using multi-part mime or some web essoterica... and then you have to build it :-( [11:17] BlueWall Slade: justin, we are fetching the same item over 60x in some cases [11:17] Andrew Hellershanks: eek [11:17] Justin Clark-Casey: bluewall: how do you know it's the same item? [11:17] BlueWall Slade: I have been digging through the packet traces and code [11:18] BlueWall Slade: at this point, I'm not sure whether the viewer is giving up and asking for the item again, or our fetch descendents is causing it [11:18]  Robert Adams: there has been an on-going suspicion that the viewer caching has problems... but it has never been traced down or confirmed [11:18] Justin Clark-Casey: I've not observed that kind of thing - just a lot of requests becaus eht viewer appears to fetch practically all the inventory [11:18] Mata Hari: you'll see in the parse I posted from my test that the 146 items that hadn't yet been fetched from my inventory were getting called over and over gain dozens of times [11:18] BlueWall Slade: we get good returs, and withing a coul\ple of seconds [11:18] BlueWall Slade: but, the item seems to never reach the viewer [11:19] Justin Clark-Casey: it's not impossible that it's a problem with polling as well, either in the opensim part or the adjustments made to the c# http server [11:19] BlueWall Slade: the http looks very clean [11:19] Justin Clark-Casey: in what sense? [11:19] BlueWall Slade: if we are the problem, I suspect that a thread is crashing or soemthing [11:20] BlueWall Slade: well, we make a request and get it back - clean [11:20] BlueWall Slade: the packet traces [11:20] Justin Clark-Casey: to what, the services? [11:20] BlueWall Slade: yes [11:20] BlueWall Slade: from the simulator to the inventory service [11:20] BlueWall Slade: about ~2 sec round trip [11:20] Justin Clark-Casey: yes,I'm begining to think that's not the source of the problem, and it is something to do with actually getting the response back to the client [11:21] Mata Hari: my sends were consistently coming back with the corresponding data within 125ms [11:21] Justin Clark-Casey: I used to think that it was a spped issue and perhaps the client request was timing out (hence this working just fine on plazas) [11:21] BlueWall Slade: it seems like it might be in the fetch descentents module [11:21] Mata Hari: but then instead of passing it on to the viewer it was re-requesting the same data again [11:21] Justin Clark-Casey: but removing the inventory lock should have doubled the effective speed but I'm not sure that had an effect [11:21] Justin Clark-Casey: bluewall: that module hasn't changed for a long time [11:21] BlueWall Slade: that seems to take a list of items to fill the skeleton and pop it in a thread [11:21] BlueWall Slade: yeah [11:22] BlueWall Slade: but, maybe some underlieing dep. [11:22] Justin Clark-Casey: the polilng stuff is pretty complex. I may end up lloking yet more closely at it in the course of trying to find out why my local behaves oddly under high bot load [11:22] BlueWall Slade: anyways, that's where I am in my search atm. [11:23] Justin Clark-Casey: although there seem to be general issues both with http and udp, which is odd [11:23] BlueWall Slade: I will look for the point that the data is getting re-requested and try to isolate it a bit [11:23] Justin Clark-Casey: yeah,would be good to know the inveral there [11:24] Justin Clark-Casey: I didn't think the viewer actually did retry the request, so that is new to me [11:24]  BlueWall Slade: each call is very clean [11:24] Justin Clark-Casey: interval [11:24] BlueWall Slade: between the simulator and the service [11:24] Justin Clark-Casey: yes, I suspect that part is not the real problem, even if it is quite slow [11:24] BlueWall Slade: ~2seconds [11:24] BlueWall Slade: that seems to be norm to OSGrid [11:24] Justin Clark-Casey: right, but you see how many requests go on [11:24]  dan banner: actually when the inv fetch does work it seems pretty quick now [11:25] Justin Clark-Casey: it's still at least a 250ms round trip for austin, at least [11:25] Nebadon Izumi: Inventory almost always fetches fast for me [11:25]  BlueWall Slade: as long as the user is in the sim, the request queue keeps rising [11:25] Nebadon Izumi: very odd [11:25] Mata Hari: thousands and thousands of requests for 150 items is the problem [11:25] Nebadon Izumi: considering how large my inventory is [11:25]  Nebadon Izumi: though when i went to BlueWalls region [11:25] Justin Clark-Casey: bluewall; it is throttled to only carry out 2 requests at a time [11:25] Nebadon Izumi: i maybe only got 5 items downloaded ever [11:25] BlueWall Slade: when they leave, it will eventually clear out [11:25] Justin Clark-Casey: up until very recently it was only 1 [11:25] Nebadon Izumi: but if i got to my regions or plazas its really fast for me [11:26]  BlueWall Slade: but, the portion between the simulator and service seems to be working very well [11:26] BlueWall Slade: I thin kwe have an internal gremlin [11:26] Justin Clark-Casey: bluewall: but you see it's quite slow???? [11:26] Mata Hari: between simulator and inventory service it isn't slow at all [11:26] BlueWall Slade: hmm, it should work as long as the request doesn't time out? [11:26] Justin Clark-Casey: yes it is, look at your logs [11:27] Justin Clark-Casey: bluewall: a particular WebFetch request can ask for contents of mlutiple folders [11:27] Dahlia Trimble is Online [11:27] Justin Clark-Casey: one such request does not translate to one inventory service request [11:27] BlueWall Slade: yes, there are plenty of "slow request" messages in the logs [11:27] Justin Clark-Casey: they also pop up on the osgrid inventory service logs [11:27] BlueWall Slade: hmm, each item is requested, in turn [11:27] Nebadon Izumi: that doesnt necesarrily mean slow network though [11:27] Nebadon Izumi: that could mean slow threading too [11:27] Justin Clark-Casey: for some reason, some of the requests take a considerable time to fulfill [11:27] Justin Clark-Casey: bluewall: right [11:28] Shez Oyen: Hi Key :) [11:28]  BlueWall Slade: in the packet dump, it looks about ~2 seconds [11:28]  Key Gruin: Hi Shez, hey All :) [11:28] Justin Clark-Casey: bluwall: so even if individual requests are quite quick, in aggregate it will take a long time to drag down a 10000 item inventory [11:28] Justin Clark-Casey: 250ms * 10000 = 2500000 ms [11:28]  BlueWall Slade: >>> That is for the full request/response including all the sync/ack [11:28] Justin Clark-Casey: whcih is 2500 secs, which is about 40 mins [11:28] Nebadon Izumi: lol [11:29] Justin Clark-Casey: bluewall: what packet dump are you looking at? [11:29] BlueWall Slade: between my simulator and OSGrid inventory service [11:29] Justin Clark-Casey: where is your simulator? [11:29] Mata Hari: he recorded me logging into his sim [11:29] Richardus Raymaker: hi key [11:29] BlueWall Slade: Mata Hara was Guinna Pig [11:29] BlueWall Slade: Hari* [11:30] Key Gruin: hi Rich [11:30] BlueWall Slade: in my home network [11:30] Justin Clark-Casey: geographical location [11:30] BlueWall Slade: it is on a pretty slow dsl [11:30] Nebadon Izumi: its kind of perplexing that retreiving a list of uuids could have so much impact [11:30] BlueWall Slade: South Carolina USA [11:30] Justin Clark-Casey: and mata is eu right? [11:30] Mata Hari: no, Canada....Ontario [11:30] Mata Hari: ping to the inventory server is ~60ms on average for me [11:31]  Justin Clark-Casey: ok, that's interesting [11:31] Justin Clark-Casey: yeah, for austin it's 250ms which is 4x [11:31] Nebadon Izumi: ya and also Pingdom who is constantly monitoring our asset server [11:31] Nebadon Izumi: for the entire past month [11:31] Nebadon Izumi: it had 100.00% uptime [11:31] Nebadon Izumi: literally 0 bad requests [11:31] Justin Clark-Casey: ok, so maybe it does make this more likely soemthing is going wrong in the simulator [11:31] Nebadon Izumi: they fetch an asset from our server every 60 seconds [11:32] Dahlia Trimble: hi [11:32]  Justin Clark-Casey: but I don't know why we don't see more reports of this issue - perhaps it's just not reported [11:32] BlueWall Slade: I think there is some speed related trigger - but OSG services are good enough [11:32] Justin Clark-Casey: at 250ms I can definitely imagine issues [11:32] BlueWall Slade: it just happens in some cases [11:32] Shez Oyen: Hi Dahlia :) [11:32]  Justin Clark-Casey: hi dahlia [11:32]  Nebadon Izumi: http://stats.pingdom.com/rcoc14tnw05o/341147?BHT-8dae65d1-03c3-4d0c-b86c-2bede2922000.7 [11:32]  Mata Hari: when I was parsing through the log I made with debug at 5 I was able to match all outbound requests with a matching response with times in the range of 125-150ms [11:32]  Nebadon Izumi: look [11:32]  Key Gruin: I've experienced the fetch hang from my region on Neb's server [11:32]  BlueWall Slade: Since I can faithfully reproduce it, I hope to isolate it to a specific area [11:33]  Justin Clark-Casey: so even if one request gets dropped, if the viewer is retrying I'msurprised it's a consistent failure [11:33]  Justin Clark-Casey: I did wonderif it could be connected with a corrupted inventory entry or something, but I think that's somewhat unlikely [11:33]  BlueWall Slade: Also, it seems that if an avatar hangs it, then a new avatar gets hung at -0- items [11:33] Robert Adams: I've got to run... take care all. RL calls. [11:34] Nebadon Izumi: here is inventory server stats [11:34] Justin Clark-Casey: bluewall: that woudl be consistent with inventory processing stopping, which would be another kind of failure [11:34] Andrew Hellershanks: Bye, Robert [11:34] Nebadon Izumi: http://stats.pingdom.com/rcoc14tnw05o/341173 [11:34] Justin Clark-Casey: bye robert [11:34] BlueWall Slade: nope, it's looking like something in the simulator [11:34] dan banner: if that was the case it wouldnt it also fail in the plazas? [11:34] Mata Hari: it's like a queue is filled and nothing new can get added [11:34] Justin Clark-Casey: bluewall: that's definitely the case? it's an interesting data point [11:34] Justin Clark-Casey: bluewall: the other possibility is that both inventory fetchrequests get tiedupwaiting for a response from the simulator [11:34] BlueWall Slade: anyways, I will continue looking at it until we can get a better idea [11:34] Justin Clark-Casey: but it sounds like that's notwhat is happening [11:35] Justin Clark-Casey: yeah, I've looked a lot at this to eliminate various possibilities and get more data but it's still elduing me [11:35]  BlueWall Slade: the part between the sim and service are working ok [11:35]  BlueWall Slade: something keeps making requests and it does make them [11:36] BlueWall Slade: but, the response is not getting back to the viewr < or the next stage of the reply in the sim [11:36] Justin Clark-Casey: ok [11:36]  Justin Clark-Casey: good to know [11:36] BlueWall Slade: I want to add more logging and possibly run in a debugger [11:37] BlueWall Slade hopes it's not in all the threading :S [11:37] Mata Hari: well, you know where to find your guinea pig when you're ready to test it [11:37]  BlueWall Slade: pk, thanks :) [11:38]  Justin Clark-Casey: It might ultimately be necessary to write a server side inventory data cache [11:38]  Justin Clark-Casey: which I hope is possible along the same lines that the viewer is supposed to cache inventory data [11:38]  Justin Clark-Casey: but that's not a 5 min job [11:38]  BlueWall Slade: how much would that add to our memory footpring in a busy region? [11:39]  BlueWall Slade: footprint* [11:39]  Justin Clark-Casey: not a lot if it's just a disk based cache [11:39]  Justin Clark-Casey: which does make it more complicated to write [11:39]  BlueWall Slade: any security issues? [11:39]  Justin Clark-Casey: seeing as simulators can access this data at any time anyway, I don't think so [11:40]  BlueWall Slade: it passses through now, but if we wright it, it's there for all [11:40] Andrew Hellershanks: Could you leverage Flotsam cache code? [11:40] Justin Clark-Casey: all who? [11:40] BlueWall Slade: that is what strikes my mind [11:40] Justin Clark-Casey: andrew: I'm not fond of that code, it has unresolved issues [11:40] Andrew Hellershanks: ok [11:40]  BlueWall Slade: haa, ok [11:40]  BlueWall Slade: oh, all == non-coders [11:41] Mata Hari: sim operator sees it but could just as easily see it by turning on high level logging and then parsing it out [11:41] Justin Clark-Casey: yes. It would just be security by 'obscurity' and not even any great obscurity [11:41] BlueWall Slade: :D [11:41] BlueWall Slade: ok, I'll keep posting about what I find [11:42] BlueWall Slade: we may just need to dot and i or cross a t somewhere :/ [11:42] Justin Clark-Casey: I'll talk to you too about what I look at/find when I get a chance to look at it further [11:42] BlueWall Slade: ok, great, thanks [11:42] Justin Clark-Casey: back in a min [11:43] Andrew Hellershanks: I have a couple of issues/questions this week. [11:43] Ai Austin is Online [11:43] Nebadon Izumi: k whats up? [11:43] BlueWall Slade: no more iussues, thtat queue is closed until further notice [11:43] BlueWall Slade: jeje [11:43] Mata Hari: lol [11:44] Andrew Hellershanks: Do region bans work? I looked at the code to check for references to the regionban table and only saw a reference in code relating to SQLite. [11:44] Nebadon Izumi: no [11:44]  Nebadon Izumi: only estate [11:44] Nebadon Izumi: the parcel bans do not work [11:44] Nebadon Izumi: pretty certain they do not anyway [11:44] BlueWall Slade: hmm [11:45] Andrew Hellershanks: ok. I found it interesting there is a reference to the regionban table but only for one DB type. [11:45] BlueWall Slade: I think group access control works [11:45] Justin Clark-Casey: back [11:45] BlueWall Slade: would htat be similar? [11:45] Nebadon Izumi: ya group access whitelist might work [11:45] BlueWall Slade: wn jcc [11:45] Nebadon Izumi: that bans everyone [11:45] Nebadon Izumi: individual parcel bans are not effective at this time [11:45] Nebadon Izumi: no one ever wired that up [11:45]  Andrew Hellershanks: ok, I can put that on my other list of things to look in to at some point. [11:46] Andrew Hellershanks: I noticed that when I was looking in to some stuff relating to the other mantis I filled. [11:46] Andrew Hellershanks: Has anyone else looked at mantis 7300 -> http://opensimulator.org/mantis/view.php?id=7300 [11:46] BlueWall Slade: the email one? [11:46] Andrew Hellershanks: I'm wondering if anyone else has ever noticed that bug. [11:46] Justin Clark-Casey: I almost wonder if viewers are trying to fetch an inventory item that doesnt' exist and we don't know how to tell them that [11:46] Nebadon Izumi: i have never experiened that [11:46] Andrew Hellershanks: Blue, no. Duplicate entries in estate tables. [11:47] Justin Clark-Casey: but then it works on plazas.... [11:47] Nebadon Izumi: oh I have seen that [11:47] Nebadon Izumi: duplicate estates anyway, but long time ago, but i dont ever recall seeing a duplicate entry for an estate manager [11:47] BlueWall Slade: never have seen it, but that is a lot of duplicates [11:48] Andrew Hellershanks: It happens once in a while. I get reports of problems in a region and that is what it turns out to be. I had to removed over 11,500 duplicates in one estate table the other day. [11:48] Richardus Raymaker: are the dub estate managers not because more sims ? [11:48] Andrew Hellershanks: I get duplicates in two different estate tables. [11:49] Andrew Hellershanks: Richardus, There are more duplicates than there are regions. [11:50] [2014/08/12 11:43]  Mata Hari: lol [11:50] [2014/08/12 11:44]  Andrew Hellershanks: Do region bans work? I looked at the code to check for references to the regionban table and only saw a reference in code relating to SQLite. [11:50] [2014/08/12 11:44]  Nebadon Izumi: no [11:50]  [2014/08/12 11:44]  Nebadon Izumi: only estate [11:50] [2014/08/12 11:44]  Nebadon Izumi: the parcel bans do not work [11:50] [2014/08/12 11:44]  Nebadon Izumi: pretty certain they do not anyway [11:50] [2014/08/12 11:44]  BlueWall Slade: hmm [11:50] [2014/08/12 11:45]  Andrew Hellershanks: ok. I found it interesting there is a reference to the regionban table but only for one DB type. [11:50] [2014/08/12 11:45]  BlueWall Slade: I think group access control works [11:50] [2014/08/12 11:45]  Justin Clark-Casey: back [11:50] [2014/08/12 11:45]  BlueWall Slade: would htat be similar? [11:50] [2014/08/12 11:45]  Nebadon Izumi: ya group access whitelist might work [11:50] [2014/08/12 11:45]  BlueWall Slade: wn jcc [11:50] [2014/08/12 11:45]  Nebadon Izumi: that bans everyone [11:50] [2014/08/12 11:45]  Nebadon Izumi: individual parcel bans are not effective at this time [11:50] [2014/08/12 11:45]  Nebadon Izumi: no one ever wired that up [11:50]  [2014/08/12 11:45]  Andrew Hellershanks: ok, I can put that on my other list of things to look in to at some point. [11:50] [2014/08/12 11:46]  Andrew Hellershanks: I noticed that when I was looking in to some stuff relating to the other mantis I filled. [11:50] [2014/08/12 11:46]  Andrew Hellershanks: Has anyone else looked at mantis 7300 -> http://opensimulator.org/mantis/view.php?id=7300 [11:50] [2014/08/12 11:46]  BlueWall Slade: the email one? [11:50] [2014/08/12 11:46]  Andrew Hellershanks: I'm wondering if anyone else has ever noticed that bug. [11:50] [2014/08/12 11:46]  Justin Clark-Casey: I almost wonder if viewers are trying to fetch an inventory item that doesnt' exist and we don't know how to tell them that [11:50] [2014/08/12 11:46]  Nebadon Izumi: i have never experiened that [11:50] [2014/08/12 11:46]  Andrew Hellershanks: Blue, no. Duplicate entries in estate tables. [11:50] [2014/08/12 11:47]  Justin Clark-Casey: but then it works on plazas.... [11:50] [2014/08/12 11:47]  Nebadon Izumi: oh I have seen that [11:50] [2014/08/12 11:47]  Nebadon Izumi: duplicate estates anyway, but long time ago, but i dont ever recall seeing a duplicate entry for an estate manager [11:50] [2014/08/12 11:47]  BlueWall Slade: never have seen it, but that is a lot of duplicates [11:50] [2014/08/12 11:48]  Andrew Hellershanks: It happens once in a while. I get reports of problems in a region and that is what it turns out to be. I had to removed over 11,500 duplicates in one estate table the other day. [11:50] [2014/08/12 11:48]  Richardus Raymaker: are the dub estate managers not because more sims ? [11:50] [2014/08/12 11:48]  Andrew Hellershanks: I get duplicates in two different estate tables. [11:50] [2014/08/12 11:49]  Andrew Hellershanks: Richardus, There are more duplicates than there are regions. [11:51] Justin Clark-Casey: I accidentally logged in the same av from another viewer [11:51] Nebadon Izumi: oh [11:51]  Nebadon Izumi: i hate that [11:51] Mata Hari: lol [11:51] Richardus Raymaker: hehe common error [11:51] Andrew Hellershanks: yea, I've had that happen [11:51] Nebadon Izumi: i got all my 80 bots logged in already justin [11:51] Richardus Raymaker: calling that 'kicking myself out' [11:51] Andrew Hellershanks: always annoying [11:52] Justin Clark-Casey: amusingly the attachments disappear on the origianl [11:52] Justin Clark-Casey: my hair suddenly went missing and I thought, huh? [11:52] Nebadon Izumi: probably moves them to the newest connection [11:52] Richardus Raymaker: i heared that attachments on HG still disappeared, at least on the dev region i where running it failed for that person. hmm. need to test it again [11:52] Andrew Hellershanks: I tried to add an index using the two fields in the two tables but it complained of a duplicate index even though the only existing index was on a single field. [11:53] Nebadon Izumi: its been like 99% better at the weekly load tests Richardus [11:53] Nebadon Izumi: it was very very bad before [11:53] Richardus Raymaker: yes neb, i stuill run version short after it where fixt. so possible mine still buggy [11:54] Mata Hari: was fun today after a crash to log back in, go back to my region and still see it sending out friends offline status notices for 2 minutes :p [11:54] Richardus Raymaker: you hav eto many friends :P [11:54] Mata Hari: yeah...oughta purge them all [11:54] Nebadon Izumi: i did an experiment a long time ago [11:54] Mata Hari: unless they send cash [11:54] Nebadon Izumi: i used to have 100s of people on my friends list [11:54] Nebadon Izumi: i purged like 75% of them once [11:54] Nebadon Izumi: and my log ins were soo much faster after that [11:55] Nebadon Izumi: that was few years back though [11:55] Nebadon Izumi: i doubt it would be the same impact today [11:55] Andrew Hellershanks: As part of fixing 7300 I'm going to have another look at my mantis 4380. I've been running mysqltuner and finding it complains about joins being done with no indexes. [11:55] Justin Clark-Casey: I think we do practically no joins though [11:56] Justin Clark-Casey: except in groups [11:56] Andrew Hellershanks: I haven't seached the code for them. Yes, groups is one place where some big joins take place. [11:56] Justin Clark-Casey: andrew: are yo uusing xmlrpc groups still?