[11:23] Ubit Umarov: Andrew's cat arrived
[11:24] Andrew Hellershanks: Hello, everyone.
[11:24] Jagga Meredith: You're late. Current topic SQL optimization
[11:24] Ubit Umarov: hi Andrew, welcome to this week opensim meeting
[11:24] Andrew Hellershanks: I was working on some software that needs to be ready for a trade show in about a weeks time. The data format got changed on me at the last minute and now I'm still trying to get the code to realiably read the data. :P
[11:25] Vincent.Sylvester @hg.zetaworlds.com:8002: I'll add the earlier conversation to the log no biggie
[11:25] Ubit Umarov: "realiably" is optimisitic :)
[11:27] Vincent.Sylvester @hg.zetaworlds.com:8002: In other news no more TryParse crashes, some issues with avsitter I cannot reproduce, so something might be awry still, but just as lost to the cause as before. At least doesn't crash, which is positive
[11:32] Andrew Hellershanks: You finished the discussion about SQL optimization?
[11:33] Jagga Meredith: I'm done
[11:33] Vincent.Sylvester @hg.zetaworlds.com:8002: Most of that is down to doing research on the current queries and how code then combines the data etc. Figuring out the "is" to plan the "should"
[11:35] Andrew Hellershanks: There are online query analyzer tools that help you optimize queries.
[11:36] Vincent.Sylvester @hg.zetaworlds.com:8002: From the few glances I had at the db connector there are not many complex ones in there so I think most of problems is code that fetches tons of smaller pieces of data and then combines them rather than letting SQL do the heavy lifting
[11:37] Vincent.Sylvester @hg.zetaworlds.com:8002: So it's down to figuring out what data it needs, how it currently fetches it and if that can be combined to a single query then to be broken up into the format it needs going forward
[11:37] Andrew Hellershanks: That seems likely.
[11:38] Vincent.Sylvester @hg.zetaworlds.com:8002: It's "get this than these things for each" vs. "get that inner joined with this" or some form of that
[11:39] Vincent.Sylvester @hg.zetaworlds.com:8002: The code itself also hasn't been touched in a while and probably has some slower methods in there or could benefit from better integration with other code and modules
[11:40] Vincent.Sylvester @hg.zetaworlds.com:8002: I only started looking a bit at that over the weekend, no deep diving yet, submarine is still out for delivery heh
[11:41] Vincent.Sylvester @hg.zetaworlds.com:8002: Hasn't been much happening otherwise for me at least
[11:42] Andrew Hellershanks: That level of deep diving is going to take time before it yields any useful information. Thank you for taking the time to do the diving.
[11:44] Vincent.Sylvester @hg.zetaworlds.com:8002: Yeah it will. Groups is one of those things that seems slow despite having no real reason for it, given the amount of data other parts juggle just fine, so something is awry in there either conceptually or just old code. It "shouldn't" be this slow even considering the data structure
[11:45] Andrew Hellershanks: Groups is one area of the code that could be doing multiple queries to get to the wanted data due to the database structure.
[11:47] Vincent.Sylvester @hg.zetaworlds.com:8002: It's programmers intuition perhaps telling me something is not quite right in there somewhere, finding it is another matter and I certainly don't want to rewrite the whole thing if I can help it
[11:48] Andrew Hellershanks nods
[11:48] Andrew Hellershanks: I've always thought the groups data structures seemed a little odd.
[11:49] Ubit Umarov: adn we do have 2 groups services in use
[11:49] Vincent.Sylvester @hg.zetaworlds.com:8002: They are not that outlandish given the data it needs, but I think it does nothing to handle it gracefully and just brute forces what it needs
[11:49] Ubit Umarov: some grids use the older flotsam one
[11:49] Ubit Umarov: like osgrid
[11:49] Andrew Hellershanks: Right
[11:50] Vincent.Sylvester @hg.zetaworlds.com:8002: I thought they moved away from that?
[11:50] Vincent.Sylvester @hg.zetaworlds.com:8002: Or was that just flotsam profiles?
[11:52] Ubit Umarov: core profiles are ok
[11:52] Ubit Umarov: they moved to them
[11:53] Andrew Hellershanks: None core profiles module doesn't use flotsam.
[11:54] Andrew Hellershanks: I just looked at the phpxmlrpc based groups code and it does have some queries that use JOIN.
[11:55] Vincent.Sylvester @hg.zetaworlds.com:8002: It's remarkable how much faster those can be compared to individual queries and joining data in code. I noticed that with ossearch getting the additional data from classifieds and such to show profile data was ten times faster in SQL compared to two queries and php doing the work
[11:55] Vincent.Sylvester @hg.zetaworlds.com:8002: Sure it puts more load on the database, but there are query caches and indexes that help with that a little bit
[11:57] Vincent.Sylvester @hg.zetaworlds.com:8002: Also a bit easier to throw hardware at SQL than OpenSim given the way things operate
[11:59] Andrew Hellershanks nods
[11:59] Vincent.Sylvester @hg.zetaworlds.com:8002: Oh right, for anyone that has taken a copy of the Hypergrid Wall I made a while back, I been working on a new version that greatly reduces the script load from 600 down to a single script. So a new copy should be available on the ZetaWorlds welcome region soon(ish).
[11:59] Andrew Hellershanks: We are at the top of the hour. Is there anyone who has a question they wanted to ask before they need to leave?
[12:00] Jagga Meredith: I'm done RL calls
[12:01] Andrew Hellershanks: ok, Jagga. Thanks for dropping by.
[12:01] Jamie.Jordan @grid.kitely.com:8002: thanks yall good meetinh
[12:02] Andrew Hellershanks: yw, Jamie.
[12:02] Jamie.Jordan @grid.kitely.com:8002: meeting :-)
[12:02] Andrew Hellershanks: :)
[12:04] Ubit Umarov: ok, rl also calls me. I was a bit quiet bc a bit indisposed rl
[12:04] Andrew Hellershanks: Ubit, np. Hope you feel better soon.
[12:04] Ubit Umarov: see u next week
[12:05] Ubit Umarov: thnx
[12:05] Andrew Hellershanks: Bye, Ubit.
[12:05] Ubit Umarov: seems osgridn groups is having issues
[12:06] Ubit Umarov: strange how that still did impact local chat and other region http
[12:06] Ubit Umarov: strange, need to take a look
[12:06] Andrew Hellershanks: Oh? Do they say what sort of issues? They have been having a number of problems.
[12:06] Vincent.Sylvester @hg.zetaworlds.com:8002: There is a mantis about it basically stopping a region dead if it takes long to fetch group data
[12:06] Ubit Umarov: failed commes on console
[12:06] Vincent.Sylvester @hg.zetaworlds.com:8002: Forgot the id
[12:06] Andrew Hellershanks: Hm... that seems a bit strange.
[12:07] Ubit Umarov: mb just a net glitch
[12:07] Ubit Umarov: also to presence
[12:07] Ubit Umarov: possible just tempory glitch
[12:07] Ubit Umarov: the strange part is how that did block local chat
[12:07] Andrew Hellershanks: That's possible.
[12:08] Ubit Umarov: shoud be o dif threads
[12:08] Ubit Umarov: but well all using the .net http code.. that may do t
[12:09] Vincent.Sylvester @hg.zetaworlds.com:8002: groups refresh isn't async it's a block
[12:09] Ubit Umarov: they had limitation lik eonly allow 2 connections etc
[12:09] Andrew Hellershanks: If the code related to those functions has not changed the next question is to ask what has changed if it isn't some temporary network issue. Did they recently update .NET, for example?
[12:09] Ubit Umarov: wel dunno.. filed on TODO :)
[12:10] Ubit Umarov: we are on mono
[12:10] Ubit Umarov: cya ppl
[12:10] Andrew Hellershanks: ok. But what version is osgrid using and did they do a recent update? Perhaps there is another bad version of mono.
[12:11] Andrew Hellershanks: With it being the top of the hour and with Ubit just left I'll close the meeting for today.
[12:11] Andrew Hellershanks: Thank you all for coming. See you again next week.
